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ABSTRACT 

Designed ^for use in programs implemented under the 
mandate of the Youth Employment- an0 Demonstration Projects Act 
(YEDPA) of 1976, this *gui-de inelhdes gui&eliaes and models for 
conducting the evaluation, research, and demonstration efforts that 
local youth staff are expected to carry on in Ntheir own local YEDPA 
programs. The evaluation, research, and demonstration activities . 
called for under YEDPA' are discussed, and guideline? are set forth 

. for their implementation. Presented next is a model local knowledge 
development case study— the Portland Career Survey Project! A 
developmental sketch for conducting program* evaluation research, * 
guidelines for applying an evaluative research framework to the 
knowledge development process, and comments on the techniques of 
evaluative research are provided. Discussed in a- section on\ ^ 
understanding national knowledge development are the learning focus 
of YEDPA and an example of discretionary project rationale. \An 

^annotated bibliography follows* (Belated youth knowledge and\ 
development reports are available separately through ERIC— s^e note.) 
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OVERVIEW 




In A Planning Charter for the Youth Employment and Demonstra- 
tion Projects Act of 1977 , the Office <?f Youth Programs arti- ^ 
culated the general principles involved in "the implementation 
Of the new youth initiatives^ The first principle was as 
follows: - • 

'.'Knowledge development is a primary aim of the new 
youth. programs. At every decisionmaking level, an 
effort must be made to ti;y out promising ideas, to 
support ongoing innovation^ and to assess performance 
as/ rigorously as possible. Resources should be.conr 

ffin+ra+^tC anr l Htmrfnrprl tM+ underlying Jidaaa _ *- 

* " can be given a reasonable test. Hypotheses and^ 

questions should be determined at the outset, with an 
evaluation methodology bu^lt in. This does not mean • 
tha€**£very youths program must be experimental , but 
rather th^t 'we should move as far as possible in this" 
f directioiV nationally and locally." 

National knowledge development objectives, supported by the N signif- 
icant discretionary authority under YEDPA,* were outlined in 
a series/of knowledge development plans. The centrally di-^ 
rected Knowledge development activities focused on those 
issues/requiring multi-site demonstrations, rigorous assess- 
ment fmd national data bases. However, it was anticipated 
that/CETA prin^ sponsors, with* the new resources provided 
under YE DP A, would try „dif ffeyrent approaches in .their lodal con- 
texts and would more carefully monitor and evaluate existing 
programs in order to see hoy( they could be improved*. Rigor- 
ous experimentation at the local level wa<s not anticipated 
because of the difficulties of experimentation 
In an operational setting and the lack of resources locally 
to achieve adequate sample "sizes. It was assumed, however, , 
that prime sponsors could increase their research and eval- 
uation activity as well as trying out some new ideas., 

To achieve this aimyv the grant package and regulations for 
Youth Employment afiQ Training Programs required prime sponsors^ 
to identify previous knowledge development activities and to 
outline plans for new ones. End-of-the-year narrative reports % 
were requested to summarize the findings of these activities. 

An evaluation of local prime sponsor YE DP A performance by "the 
National Council on Employment Policy concluded that some sig-\ 
nif icant activity was generated by this mandate: 
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'Under the* gun to do something, most sponsors responded. • 
Some repackaged evaluation work they had already been y 
doing and called it knowledge development. Most made, 
an attempt at some kind of explicit evaluation experience; 
spme were quite' elaborate. A few got away with doing 

- nothing. .On the balance, though, the* research ajid ~ v » 

analysis mandate produced a surprising amount* of new, 

• local activity.!... Wh^t is especially promising* is tHe 
fact that in the second year of the yoj|||K programs , local 
sponsors^ still, lacking specific /latio^^gof f ice^ guidance , 
* took the initiative in judging *the me^itlrof their know- 
ledge development projects and frequently completely 
revamped them." 

The NCEP evaluation noted al so._j diat; 11 p r i me-_-sportsar^-Iraugi--hg^n _ 
almost entirely on their owifswhen it has* come to developing 
research, demonstration or evkluatign initiatives, or testing 
innovative prograih^ideas . There was very little guidance 
from the Office of Youth • Programs on this issue, in contrast 
to the carefully structured nationally-directed knowledge 
development efforts. % j 

Evaluation Research In Local Youth Program Development - provides 
such direction for prime sponsor activities. It is, in essence,, 
a •framework for local knowledge development planning and. imple- 
mentation. Concurrently, there has been an evaluation of local 
knowledge development .activities to identify models, and an 
effort to establish technical assistance capacity to provide 
detailed hands-on .guidance to interested prime sponsors. This 
monograph describes not only what prime sponsors can initiate^ 
locally, but ho\$ they can" utilize the products of nationally- 
directed knowledge .development activities. 

***** 
This volume is, itself, one of the mSny products of the "know- 
ledge development" effort implemented under the mandate of: the 
Youth Employment and Demonstration Projects Act^.of 1977. The 
knowledge development effort consists of hundreds of separate < 
research,, evaluation and demonstration activities which will 
result in literally thousands of written .products . The'activ- # 
ities have beeh structured from the o l utset so that each is 
self-standing fc>ut -also interrelated with a host of other activ- 
ities. The framework is presented in* A Knowledge Development 
Plan for the Youth Employment and Demonstration Projects Act of 
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Information is available' or will be coming available from- the . 
various knowledge development activities to help resolve an 
almost limitless array of issues, but 'answers to policy' 
questions will usually require integration and synthesis ' . 

from- a number of "separate products, which, in turn, will < 
depend on knowledge and availability of these products. A 
major shortcoming of past research \ evaluatibn # and demon- 
stration activity has been the failure to organize and dissfem- 
iriate the products adequately to assure the full exploitation 
of the findings. The magnitude and structure of the youth 
knowledge development effort puts a premium on organization 
and dissemination. ' 

As part of its knowledge development mandate, therefore, the, 
Office of Youth Programs of the Department of Labd* will orga- 
nize, publish! and disseminate the written products of all major 
research , evaluation and demonstration activities supported 
directly by or mounted in conjunction "with T;Ke~kriowle"age 
development effort. Some' of the same products may also be 
piiblished and disseminated through other channels, but they 
will be included in the structured series of Youth Knowledge 
Development Reports in order to facilitate access and inte- 
gration. J" ' 

The Youth Knowledge pevelopment Reports , of which th;Ls is one , 
are diyided into twelve brd|td categories: _ • c 

1. " Knowledge Development Framework; The products in this 
category are concerned with the structure o£ knowledge develop- 
ment activities, the assessment methodologies ^hich are employed 
validation of measurement instruments, the translation of know- 
ledge into policy, and the strategy for disseminating findings. • 

2. Research on Youth -Employment and Employ ability Develop- 
ment : The products in this category represent analys s of exist- 
Tng~data, presentation of findings from new data sources, / 
special studies of youth labor market problems' and policy 
analyses . ■ ~ 



1 ' 3. Fr'ogram Evaluations : Tlte products in this category L 
include impact, process, and benefit-cost evaluations of youth 
programs including^the Summer Youth Employment Program, Job 
Corps, the Young Adult Conservation Corps/ Youth Employment and 
Training Programs, Youth Community Conservation and Improvement 
Projects, and the Targeted Jobs Tax Credit. \ * 

. 4 Se rvice and Participant Mix : * The evaluations and # 
demonstrations summarized in this category concern the matching^ 
of different types pf youth with different service combinations. 
This involves 1 experiments with work vs.. work: plus remediati'on 
vs. straight remediation as treatment options. It also^mcludes 
attempts to mix disadvantaged and more affluent participants, 
as well as youth with older worKers. • 

t 



5^ Education and Training Approaches t The products in 
this category present the findings of structured experiments 
• to test the impact and effectiveness of various education and 

# vocational training approaches including specific education 
.methodologies for the disadvantaged, alternative education, 

approaches and advanced career training. 

• 6. ^re-Employment and Transition Services : The products 
in this category present the fillings of structured experiments 
to test the imp t act and; effectiveness o*f school- to-work tran- * 
. sition activities, vocational exploration, job-search assis- 
tance and other efforts to better prepare youth for labor market 
success. 

7. < Youth Work Experience ; , The products in this ^^tegory 
addrfess the organization of wcgrk activities, their output, pro- 
ductive roles for youth and the impacts of various employment 
approaches. , 

8. Implementation Issues ; This category includes cross- 
cutting analyses of the practical lessons concerning "how-to- 
do-it. Issues such as learning curves, replication processes^^ 
and 'programmatic "batting averages" will be addressed under 
this category, as well as the comparative advantages of alterna- 
tive delivery agents. 

9. Design and Organizational Alternatives ; The products 
in this category represent assessments of v demonstrations of 
alternative program and delivery arrangements such as consoli- 
dation, year-round preparation for summer programming, the ufe 
of incentives and* multi-year tracking of individuals. 

10. Special Needs 'Groups ; The products in this category 
present^ findings on the Special problems of and adaptations 
needed *f or significant segments" including minorities;* young 
- mothers, troubled youth, Indpchinese refugees and the handi- 

• capped. 

* . * 
11.. Innovative Approaches ; The products ix\ this category 
present the 'findings of those activities designed to ' explore 
new approache's. The subjects covered include the Yout£ 
Incentive Entitlement Pilot Pro jects, .private sector initi- 
atives, t>he national youth ' service experimeryc, and energy 
initiatives in weatheriz^tioh, lpw-head hydroelectric dam 
restoration,, winflpower and the like. 

% 12. Institutional Linkages : ^he products in this category 
wili include studies of institutional arrangements' and linkages 
y as well as assessments of demonstration activities to encourage 
such linkages with education, volunteer groups, drug abuse 
. agencies and th.e like. ' 
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In each o£> these knowledge development categories, there will 
be -a range of discrete demonstration, research and evaluation 
activities, focused ozi different policy, program -and analytical 

V issues. For instance, all experimental demonstration projects _ . 
have both prooess and impact evaluations, frequently undertaken 
by different evaluation agents. Findings will be published as 
they become available so that there will' usually be a series 

«*• of reports as evidence accumulates. To organize these pro- • 
' ducts, each publication is classified in one of the twelve 
broad knowledge,, development categories, described in terms of . 
the more specific issu£', activity or cluster of activities to- ^ 
which it is addressed, with an identifier of the. product and \/ 
what it represents relative to other products in the demon-- 

stfationT" Bence-, tfte multiple products- under— acknowledge- 

development activity are closely interrelated and the activ- 
. ities in each broad cluster have significant interconnections. 

This framework for local knowledge development activity is, 
closely related to aTl of the products in the' "knowledge 
development framework" category. If should' also be 
assessed in conjunction with Youth and the Local Employment 
Agenda' in the "program evaluation" category as' well as 
Local Mixed Income Testing in the "service and participant 
mix category" which assesses the success and problems prime 
sponsors have had in' pursuing the knowledge development 

mandate. . ' 

f i t 
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. . CHAPTER I 

, WHAT IS KNOWLEDGE. DEVELOPMENT? ' * ' 

c ' 

x • - % • 

Understanding the KD Challenge v 

A primary emphasis *bf the youth employment and demonstration projects is on 
learning about the mature of employment problerr^s among youth, and about the^ 
appropriateness and effectiveness of a variety of strategies for dealing with these 
problems. The hope of DOL 4 is that a few years of well-documented research, 
evaluation, and demonstration projects wilt yield information i or making programs 
more effective in the long run. * f ^ 

r 

A 

This learning cpmponent of . the Youth Employment Demonstration Projects Act 
(YffoPA) was dubbed "knowledge development" at *he inception of the Act. The first 
of tefpprinc^ in the YEDPA planning charter stated that a principal objective of 
YEDPA is to develop knowledge from the youth initiatives about how to best assist 

youth. * ' 

» 

« « 
It is crucial for local youth staff to realize that KD has a dual foous: the national 

level and the local level. At the national level, the Office of Youth- Programs (within 

the .Employment and Training Administration) is overseeing a broad, structured array 

of multi-site demonstration projects, large scale evaluation's, and .coordinated research 

, efforts, all funded with discretionary funds. , « 

\ 

The largest of these discretionary projects is the Youth Incentive Entitlement Pilot 
» 

Project (YIEPP), for which 17 prime sponsors across the country were selected to 
operate, very structured pilot projects testing the effects of guaranteed jobs on school 



dropout rates and youth employment behavior. Although directly funded and directed* 
by the national office, most of the over 200 demonstration and evaluation activities 
were conducted, at local prim^ sponsor sites. The results of these finding^ will be of* 

4 M 

great importance to prince sponsor youth staff. As of FY'80, a principal focus of the 
Office of Youth ^ograms is to synthesize and disseminate the results of discretionari- 
ly funded projects to prime sponsors. National knowledge development organization "is 
farther discussed in Chapter 4. 



The other aspect of v knowledge d^yelopment activities, that which is emphasized. in 
''this guide, is the evaluation, research, and demonstration effort all local youth* staff 
are expected to carry on in their own local youth programs. DOL hopes, on the" one 
hand,^that these KD findings will benefit individual local youth staff immediately and 

directly by increasing their understanding of the youth employment problems and 

... • L 

programs in their respective areas. On the other.hand, it is also hoped that these local 

* * w. • 

knowledge development efforts win encourage lockl administrators to institutionalize 
processes foi\ assessing youth programs and utilizing evaluation finding^-lfTnew local 
policy formulation. However, it is important to note that these hopes for local 
knowledge development efforts stem primarily from the federal perspective of what 
prime sponsors should be doing. In fact, local youth staff s' int£r£i^tations of the 
value, functions, >^nd purpose of local knowledge development have often varied 
considerably from that of the national office. A brief examination of prime sponsor 
reactions to, and experiences with, KD in FY 78 ^nd FY 79 may assist the reader in 
grasping the principal pitfalls, as well as promises, of local knowledge development. 

Two of the most important sets "of realizations^about local conduct of knowledge ' 
developmenj activities Concern making methodological and locally relevant decisions. 
As a general rule, less elaborate, basic approaches to youth prpgram assessment often 



provide more fruitful results than the use of more sophisticated approaches. In FY 78, 
many prime sponsors and subcontractors interpreted knowledge, development as a 
mandate to' do elaborate social experimentation. Such experimental social design 
often included experimental arid control groups of participating' and non-participating 
youth." Such appVoaches often included the^lO% mixed income test, which examines 
the notion that mixed income participant backgrounds provides for improved youth 
outcomes. Geherally speaking, these more sophisticated experimental approaches to f 
assessing youth program outcomes were not successful. This is not to say that some 
good ideas were not Being ' inspected, bGt.trfct the manner of ' inspection was often 
impropriate and did riot work." For example, youth staff found that in,tJecentralized 
balance of state prime sponsorships, it was not practical to set' up tight experimental 
conditions. More cHhtralized prime sponsors, on the other hand, discovered the 
difficulty of maintaining suff icient^coj)trol group size. Youth staff also found that 
experimental models were often.awkward and unrealistic because staffs were not able.* 
to control, or even adequately measure, non-income variables. Then ,too, youth 
planners found that findings from the experimental approach were often too late to- be 

politically useiul. • * ' 

\ ' - 

Learning from their mistakes, many primes and subcontractors took simpler, more 

basic approaches to youth program assessment in FY 79. Less elaborate approaches 

" ' ' i • 

often proved to be more manageable, more likely to provide locally useful information, 
and to serve as building blocks for expanded and/or slightly more sophisticated 
projects. ^ • ' 

A second, related lesson of knowledge' development for local youth staff in FY78 and 
FY 79, is the importance of focusing local youth program evaluation interests on 
variables under the control of the local prince sponsorship. A local learning focus 



presides year to year continuity and cumulative progress in local knowledge ^evelop- 
ment effbrts. By contrast, l§ss fruitful results were provided by local knowledge 
development efforts investigating questions mainly academic interest or broad 
policy issues, which could principally be settled only at the federal level (e.g.. 10% 
mixed income tests), ' * 

*■ ' — ' v 

Benefiting'f rom these lessons, this guide emphasizes the utilization of relatively more 
straightforward approaches to youth program assessment and evaluation, although it 
> also contains advice and instruction in conducting more sophisticated experimentation. 
In particular, the guide presumes that the principal ' reason for local youth staff to 

, 6. * 

conduct knowledge development activities is. for the purpose of supporting locally 
initiated youth program policy change. The guide f s message is equally applicable for 
youth employment and training program, staffs in CETA prime sponsors^ school 

districts, CBOs, various LEA and CETA subcontractors,- and similar agencies, ' °" 

** * f* • *■ • * 

The model approach to knowledge development, discussed in the following section, is 

based on many observations of how local youth staff have successfully approached the 

learning cycle of youth program planning and design, program delivery, assessment, 

and redesign. The general approach to KD described here places highest priority, on 

the development of 'realistic and useable information for local policy purposes, and 

suggests that the choice^ of specific evaluation tools and approaches be largely 

! i 
dependent on how much is currently known about -the specific local program being 

assessed, a>$ well as on what iyp%ct}£al with given resources.and expertise. * $ 



Toward A Working Understanding of the Knowledge Development Process 

A general model which synthesizes sind summarizes the approach that many local 
youth staffs have successfully used is presented and discussed in ,this section. An 
understanding of this- model at the outset should give the reader a general background 
of the knowledge ^development process, as well as a feel for key decision points along 
the way. A case study example^f a prime sponsor's utilization of this knowledge 
development motiel through several program development cycles of a particular youth 
program i^ubsequently presented in Chapter 2. Some readers may wish to consider 
this specific" case study prior to examining the more general presentation of the basic 
KD approach as discussed in this section. In either case, it should be recalled that the 
model presented here is based on analysis and observation of the ways many local 
youth staf f s 'have successfully approached learning from their youth programs. The 
basic "approach which emerges rrjay be utilized on many different levels, from the 
simplest* and most straightforward youth program development efforts of very small 
rural prime sponsors or subcontractors to the most sophisticated and research-oriented 
ones of large unban primes, LEAs and CB'Os. The basic logic of the approach remains 
the same. 



At the level, knowledge development is probably best understood as beifSfc— - 

synonymous with what most youth coordinators think o^as "youth program develop- 

i 

"ment." As shown in Figure 1-1, the basic program development process usually 

* 

consists of: , N 

1. Planning and design: Identifying the problem, setting goals for problem 
solutions, examining alternative solutions, and designing a program of 
youth services ~ 

2. Implementation: Implementing the program of youth services 
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FIGURE 1-1: THE PROGRAM DEVELOPMENT PROCESS 
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3. Assessment: Monitoring and evaluating the ^iplementation and results of 



t * . the youth services program 



4. 1 Feedback: Applying the results of monitoring and evaluation findings back 
into the original problem identification and allowing for subsequent jnodif i- 
cation. The' cycle .may then be repeated, though at a more developed level 

of experience (or knowledge base)/ Learning has occurred. 

4 * < - ? 

In knowledge development, considerable emphasis is often placed on the program 
evaluation component of the development cycle. Even so, it is crucially important to 
understand that all the elements of the local youth program development <^cle must 
fit together as parts of a whole. Thus, the program development elements must link 
and interact in a compatible way. 



For example, it would not normally be wise to design and demonstrate a new career 
awareness program, then narrowly evaluate the implementation phase by experimental 
comparison of precise participant test scores of some sort. In such a case, our 
evaluative approach would be "out' of sync" with the developmental level of our 
program- planning'design and implementation. Instead, we might prefer to evaluate the 
demonstration implementation by using simple quantitative measures (planned vs." 
actual' enrollments, terminations, placement rates, expenditure rates, etc.) along with 
qualitative indicators (interviews with counselors, participants, others). 

The cumulative learning approach to youth program development modeled, in Figure 1- 
-2 emphasizes the importance of focusing on locally relevant program questions, as well 
•as appropriately tailoring % assessment style to the stage of program development. 

Youth planners, typically, are immersed in an ongoing system of program development. 
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1 

However, for purposes of understanding the model KD approach, we can break into this 
continuous cycle of program development at the point labeled in Figure 2-1 as 
"Current Youth Program Assessment/ 1 Here we are examining the findings of how 
well our present y$uth programs are operating, their cost effectiveness, how well they 
are being received by\youth participarits, an<}, as far as we can determine, the impact 
these programs are having on participants. 

• « 
Next, we analyze the local youth labor market, asking ourselves such questions as: 
Are the youth programs being offered addressing current youth needs in'the communi- 
ty? Are changes in the yoOth community, regulatory changfes in target grpup 
identification, or other circumstances creating new needs and, perhaps, new un- 
knowns? 

Usually problems or questions we need more understanding about frill e jP er g e fr0m tl } is 
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effort to clarify and focus our youth program offerings. Here is the raw material oat 

of which we can begin <to identify the knowledge development focus which is "right 11 

,ior our jurisdiction Although the focus area stems from our assessment of current 

, • offerings and needs in local youth programs, we will, of course, bring together a broad 

array of external tools and resources to meet this challenge. For example, we may 

"pull -upon past experience in our own staff for youth programming*. Each-experienced 

youth prograrrTstaff member involved in the planning effort will most likely draw upon 

experiences and lessons learned in somewhat similar situations elsewhere. Then too, 

there is the ever changihg, ongoing sense of understanding ab^ut what works and what 

does not work in youth programming which is shared by all employment and training 

youth program staffers. Although this "invisible college; 1 or body of professfonal 

understanding, thrives and is communicated in-a number of ways, pro^ly the most 

important method utilized by youth staffers in obtaining this information is informal, 
v 

■ ~ ■■ / . . 

* . t 



* 



communication with colleagues in other jurisdications or \>cations. A "Knowledge 
Development Interest Network" now exists to help energize this ongoing, evolutionary, 
network of understanding about what does and does not work in youth programming. 
By tapping into this body of professional understanding when identifying our own local 
youth program needs and knowledg^ development focus, we. can avoid re-ihventing the 
youth program^ wheel. ♦ 1 

In reviewing, prioritizing, and selecting our specific KD focu^we may find it useful to 
distinguish between questions of "program" focus and "variable" focus. That is, are we 
^ principally interested in the ptirely practical outcomes of a specific program at a 
, specific time ( program questions), or are we more interested in' singling out specific 
components of the program as more generalizable stimuli? In other words, i£ we can 
focus upon learning about the general variables underlying a program (rather, than upon 
the effectiveness of the programjas a whole), we mayf>roduce findings of greater local 
significance. This point is especially relevant tq learning f ro'm demonstration projects. 

An example m^y help illustrate this point. Suppose we wish to learn about a youth 
career information center demonstrated in a lo<pal high schteL Measuring the number 
of center users (for example') will be necessary' for reporting purposes, but may help ys 
learn only about this specific center. However, if the center" ; is set up to test some 

specific principle or variable in youth career center establishment, say, the relative 

,** * 

effectiveness of personal versus formal appeals for attendance; our results may have 

m ^ 7 . ■ • . / 

greater transf erability/o other local career center situations*. 

" L - ' ■ : '.■ \ ' 

*Thus emerges a possible difference between "Knowledge Development" and ^program 
Evaluation". The' development of relevant local knowledge-tends to stress Continuity 
and cumulativeness of youth program findings, while purely conventional program 
evaluation is often a one-shot proposition. The difference between the two is found in 
the> rationale, focus, and definition of what we wish to learn, rather than in the 
specific methods and techniques of study. ' 

• » • . 
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Assuming the, identification of A realistic and locally relevant learning focus for our 
youth programs, we must consider how much we actually know abo^t this particular ' 
local KD concern. This important consideration will obviously have a great impact on 
the way we approach designing the program of youth services to be offered, as well as 
the way these services are actually delivered. For example, e if we know very little 
about a new CETA-LEA linkage we would like to establish, it is likely that our first/ 
order of business will be to select a program .model or approach to the possibility of 
this new linkage. The source of this program model might be our unique local insight 
into local needs, or* it might represent an "imported" idea, or variation of an idea 
borrowed and adapted from another prime sponsor.* Next, we would probably design 
and implement a model demonstration of the CETA-LEA connection we wish to try 



out. t 



'it may be that we have Already gone through an original program development cycle 
demonstrating this new institutional linkage, in which case, it is likely that unless the 
demonstration was clearly unsuccessful, we will be interested in running a modified 
version of the original model or approach. This- modification will incorporate what we 
have T!fSrned from the initial model -demonstration. The plafining-delivery-f Redback- 
modification cycle will continue, each cycle adding to our understanding, of the 
evolving CETA-LEA approach that works best for us. As our understand^ becomes 
more and more complete, we ar.e able toask and answer more and more -specific 
questions about the^nature of what Works in our CI?TA-LE:A coordination eff or} and 
what does not. m 

■ J. * • 4 - ' v 

* Exportation and importation may occur in vaAous ways: e.g., through the Knowledge 
Development Interest Network, through DOL Regional federal representatives or 
youth coordinators, through tMe Office of Youth Programs, through, prime sponsor 
associations, through the youth program "information broker" described in Chapter 4,. 
or through more informal professional exchange. 



Vhus, at each program development stage, we consolidate and apply what was formally 
and informally learned f 150m the last development pound, fihding ourselves at a sort of 
frontier $r margin of^ understanding about the particular situation. The .next bite, or 
increment, of unkrfown area to be explored at this frontier' cbntains future candidates 
fQr knowledge 'development which will be considered and prioritized for our next round 

r * • 

of program development. We move from a relatively ^ general to an increasingly 
particular and complete development of understanding about the particular youth 
program being considered. ... f i 

x • ' ' ' 

Identifying the type of knowledge development inquiry which is appropriate for a 

prime's specific yodth program demands an accurate understandings of how much is 

collectively known locally ^bout the particular program approach at the. time. The 

njore we* know about the program, the mc^re specific and precise we car! be in the 

hature of the qufestions we, ask and answer about^the pjggram. In some ways, this 

question fitting process is comparable to deciding how "far 6ut the ice you" should 

'walk oh a newly frozen pond. As you walk closer to the middle of the pond, you leave 

* p * \ < 

more solidly frozen ice and risk a dunking. So too, if we^attempt to ask and answer^ 

overly Sfecific questions about a youth program prior to developing a firm understand^ 

ing of how the process works, we risk procedural failure irvBjjr analysis or, at least, the 

• a ^ ^ 

possibly of collecting information \yhich is erroneous or useless.* 1 * 

f ' * » 

In Figure 1-^ eight sequential levels of evaluative focus are identified which 
correspond or are parallel wjth a general spectrum of progra/n development. Although 
the sequenq^f^ay be reorderd to fit specific local circumstances, the progression 
shown depicts a logical,- essentially chronological order of assessment -foci. For a 
given developmental stage of a program, it is likely that a local youth staff will find it 
appropriate to orient its assessment around some point along this^spectrum. Positions 

12 22 



shown in Figure 1-2 include: > 

1. Program identification or creation. Assessment * of c£ model program 
: 1 " : r 

* approach which has either been internally generated or modified from an 
approach used elsewhere. 

2. Program Model Demonstration . Process evaluation of initial model demon- 
stration. ^ , 

3. Program ModeTModif ication. Predominantly a process oriented evaluation 
of a* modified version of the original demonstration model. Some simple 
outcome measures may begin to be used at this poirvt. ^ 

4. Process Evaluatipn of the Model. Possibly done separately from Step 3, if ; 
circumstances warrant a general progfam implementation assessment. If 

included,jodtcome measurement is still simple and de-emphasized; but it is . 

, — t 

j likely to be of increasing importance and credibility. 



/ 
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* A process study provides information about the mean^ ggiployed -tti a program to 
produce specified outcomes. Especially important during the developmental'phases of 
a model or program component, proqess studies are often / employed before the 
program as a whole can be described. Thus^ process stucfrej( of program. implementa- 
tion are com/non, examining, for example, attendance rates, component completions, 
staff -participant ratios, rates or types of attitudinal change, termination by reasop,, y 
etc. Process studies can not only help develop the program, but they also help to 
conceptualize what the program is and' how it works. t 

An outcome study describes the results of the program process on program partici- 
pants. Outcome studies are typically concerned with producing a summary statement 
about the effectiveness of the program or some phase of it. Both the audien^and 
tiding involved in outcome evaluations will be different than in a process study. 

, »' * ■ ' * 

Impact studies try to measure the overall effect a program, including the 
achievement of goals in behalf of the target group, as well as the diff icult-to-measure 
side' effects of the program on the general population or community. Some evalyator's 
terminology doesn't distinguish between "impact" and "outcome" measures, typically 
lumping both under the term "impact." 

An evaluation study often blends outcome and process. It may be useful to have 
process data to assist in the interpretation of outcome results; so, too, it may be 
Appropriate to incorporate an outcome measure into a» predominantly process oriented 
study. Nonetheless, Jthe basic orientation of an evaluative study as a whole is like to 
fall into jane category or the 'other ♦ 
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5. In^Prograrg Assessment of ih$ Model . Evaluation of the .interactional 
effects bet\Jeen the study progrifc component or program and other, on- 
going, components or programs. May be part of Steps 3 or *f. Example: A 
new intake approacfrlras been tested "in isolation 11 in earlier steps of the 
sequence; now we concentrate on the "ecological" effects of the new 
approach in the total program environment. 

6. Outcome Evaluation of Participant Benefits — At Program Termination: 
Ideally, we now have a rich process" understanding of the -program 
approach as it has developed. We now begin to focus pretiominantly<on the 
measurement of outcome effects of, the program on its participants, though 
we continue to make 3 process observations. Our process/outcome balance 

x has reversed its position from that in Steps 1 and 2. Rather than just 
' drawing inferences, we may now wish to move toward answering program 
outcome effectiveness questions in terms of how sufe we are that our 
^program actually causes the outcomes we claim. ^ 

7. Qutcome » Evaluation of Participant Benefits — Longer Terrrf^Benefats : 
Similar to Step 6, but involving f ollaw-up of participants over some period. 

V Impact Evaluation of Effects on the Youth Community: Our focus turns to 
— ^ • . . 

estimating or measuring the "ripple effects" of our program on the larger 
local youth community or- thfe local geographic community of which the 
program particigants are part. 

Within a "single program development cycle, youth staff may choose to focus .their 
learning efforts on more than on6 of these levels, simultaneously or sequentially.. An 
obviousi limitation on the scale of the learning eff oct is resources — time, money, and 
expertise. However, it is important to realize that an inherent, "horse before the 
cart" logic acts as a delimiter as \yell. For example, it generally doesn ! t make sense to 
narrowly measure career guidance outcomes prior to first answering such questions as 
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"Does our new program include career guidance at all?" 

The rate of progression through the learning/assessment spectrum will also b,e 
delimited by local youth staff resources and by the practical length of a program 
(jdeyelopment cycle. For example, two-week cycles of intense remedial reading may 
pass through* multiple developmentflearning cycles' a more elaborate program of 
services passes through only one cycl^. 



3ust as clarification of bur program's stage of development helps determine the proper ■ 

7 

assessment stylejtateng^the process-outcome evaluation spectrum), so, too, goes this 
clarification have implications on our choice of assessment methods. 



As will be discussed*more completely In Chapter 3, process oriented levels of inquiry 
are often most appropriatley approached using predominantly qualitative, subjective, 
and often judgmental methods of analysis. However, this is not to suggest that simple 
quantitative data 'should not be collected att this early stage. For example, quarterly 
characteristics summaries and basic MIS data will always be collected for any program 
and will, typically, be of great use in process evaluations. More precise types of 
quantitative measures (such as ^arious test results) are necessary to answer outcornte 
evaluative questions* Thus, *p those levels of inquiry where our principal focus is on 
outcome or impact jmeasurement, we will predominantly be interested in collecting 
preciseyquantitative information — although even then we may continue to have an 
interest in collecting at least some qualitative information to provide a context ^of 
understanding for^our more precise quantitative findings. 



Regardless of which particular integration of program development stage, assess- 

c 

ment/learning focus, and method, orientation makes sense for a given youth program, 

youth staff will generally apply a basic evaluative research approach to come up with 
* • 
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their findings* This evaluative research logic, further discussed in Chapter jf, involves 
goal analysis, methodological considerations, a research plan or design, and document 



tat ion. 



The "key to the entire local knowledge development "effort is in the utilization of 
results. If assessment reslilts are not utilized, th.ere is very little reason to obtain 
them. Much of the remainder of this guide more specifically explores the application 
of assessment Methods to the systematic aspects of learning from local youth 



programs. 
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CHAPTER n 

A LOCAL KNOWLEDGE DEVELOPMENT CASE STUDY: 
THE PORTLAND CAREER SURVEY PROJECT 
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CHAPTER 2 

A LOCAL KNOWLEDGE DEVELOPMENT CASE STUDY: 
THE PORTLAND CAREER SURVEY PR03ECT, 

< 

This case study was selected to illustrate how the knowledge development process 
occursy Althgjigh the composite of locally generated knowledge development subject 
areas is vast, this case study represents one* prime sponsor's attempt to^focus local 
learnir)g efforts on a single program category to determine the issues*to concentrate 
on, design an approach to deal with the identified needs, and demonstrate, assess, and 
continually refine, the approach. 

The case study is representative of the youth program development process because it 
illustrates how the insights and knowledge gained from each program cycle are used to 
continually modify the program approach so that it better meets the needs of the 
target population, and because it adds to the local youth staffs 1 understanding of which 
approaches work -best for which groups of youth. The case study illustrates "how a 
prime sponsor, in its attempt to upgrade^programming and be responsive to the 
dynamics of youth employment and training works'through the knowledge development 
profiss (Figure 1.2: A MODEL OF THE LOCAL KD PROCESS). 

The ^rationale for utilizing a "case study is based upon the reality that specific, 
observable processes and stages of program development do occur iri the real world of 
youth employment and training, and that those processes occur systematically and can 
be captured, documented, and utilized to further enhance the effectiveness of local 
youth programming. 

The case study depicts a chronological progression in program development and scope 
and describes the various phases of the local knowledge development process. Specific 
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emphasis is placed on the elemenjs of youth program assessment. The importance of 
focusing on process-oriented questions during the early stages of program development 
and implementation is exemplified, as is_the appropriateness of focusing on outcome 
and impact questipns during the latter stages of program development. The usefulness 
of qualitative and quantitative assessment techniques in seeking answers to "how a 
program works and why" knd "what types of benefits or impact this program has on the* 
youth participants" is highlighted. The purpose is to show hdw low-cost, simple, and 
effective assessment methods can be combined with easily gathered numerical data to 
tell us what is working well within a program and which program elements are not 
bringing about the desired results. The case study also underscores the vital role of ? 
"goals analysis" in moving through the knowledge development cycle in an orderly and 
systematic way. 

The Portland Career Survey Project was selected because of the availability of 
information and documentation with which to track the knowledge development 
process .from the program's, inception (1974) to its current state. In addition, the 
program has passed through several iterations of the KD Mode^ Process' and has been 
replicated in similar settings. Moreover, the Portland Career Survey Project repre- 
sents a prime sponsor's practical, although imperfect, effort to learn from, develop, 
and refine youth programming strategy in a low-cost, effective manner. \ 

4 
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PROJECT PROFILE 

The Career Survey Project is a career awareness program designed, for 14 and 15 year- * 
* old economically disadvantaged youth*, The Project, designed by the Cit>; of Portland^ 
Oregon, Prime Sponsor provides hands-on career experiences and career ^information 
through, direct exposure to 10 career cluster areas: office occupations, small engine 
repair, electronics, data processing, communications/broadcasting, food services, 
patient care, graphic erts, construction technology, and engineering* The project is 

c 

O 19 

E^C - . 20 

JL 



\ 



- / 

operated during the summer in a community college setting. 

The project serves youth who normally do jlot take advantage of the career education 
training available in public high schools. By identifying these youth and giving them 
the opportunity to earn money while being exposed to aspects of employment in 10 
career areas, it is anticipated that they will be better able to make career and 
educational decisions aff eoting their futures. 

Features of tfae Career Survey Project 

. *- c 

o Allows a prime sponsor to create well-structured, substantive surhmer career 

exploration experiences for 1*-15 year-old youth 
o Youth like it >, 
o Low-cost; easy 'tp administer 

o Provides youth with career experiences fulfilling state Ind local academic credit 
requirements 

o Effective for youth with short attention spans and basic skill deficiencies 

o Easily replicated (the Project has Seen replicated at two community colleges; 

the project, design was used as the basis for a universjrty-based knowledge . 

development project (Uryversity of Oregon Medical Careers Exploration Project) 
o Lends itself to replication in industrial and business environments 
o May be adapted to serve speqif ic client groups 

o Project structure .easily accommodates joint sponsorship ^across' prime sponsor 
jurisdictions (Summer Youth Employment Program (SYEP) youth from two prime 
sponsors areas participate in the«project on a cost-share basis.) 

o High school counselors are supportive of the project; recruitment and academic 
credit provision are facilitated by such endorsement * ' 
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Prime Sponsor Description; City of Portland, Oregon 

Portland is a city of 385,000, the center of a metropolitan area containing 1.1 millidn 
people. C£TA is administered by the City Human Resources Bureau; Youth CETA „ 

% 4 * 

t 

services .are administered through a network of youth career training centers. These 
City operated training centers provide comprehensive services for^ut-of-school youths 
and administer programs for in-school youth. Services to I/S youth are provided via a 
non-financial agreement with Portland Public Schools. Formula youth CETA dollars 
allocated in FY79 amounted to $3,245,000. 

-.. •• . , ' 7 - 
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Logistics of Project Operation 

■ The Project is operated via contract between the City of Portland Prime Sponsor, 

✓ - \ 

Portland Community College, and Portland Public Schools. Portland Community 

■ College conducts the project; Portland Public School assumes advisory and youth 
referral responsibilities. ^ 

TV 

The. project, operative since 1974, has undergone five modification cycles, each cycle 
providing a slightly different focus and level of refinement. However, the core design 
involves the rotation of youth through career awareness and exploratory classes in 10- 

. career cluster areas. Community College instructors design the curriculum around the 
career information and participatory (hands-on) focus of the project, and youth 
(grouped in teams) rotate from one cluster area to another on a weekly basjs under the 
guidance of a team leader. Each youth spends 3 hours per day in career exploratory 

. classes and 3 hours'per day in supervised work activities. 

v ■ , ■ 

c 

The adult team leader serves multiple functions. The team leader monitors the team's 
. flow through the*5ystem, supervises team work activities, acts as an instructional aid 
to the career cluster instructor, maintains participant time and attendance reports, 
and provides counseling to youttvparticipants. 
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Initia£uy, in r974, the proj^:t served 50 youth. As time progressed, the Prime Sponsor 
and school District recognized the value of the summer career exploration concept 
, and took measures to refine the project and axtend the experience to increased 
numbers of SYEP youth. , The following table (2-1) reflects the growth factor (increase 
in participants served) as well as the characteristics of the participants. Figure 2-2 
'reflect^ the*cosft of the project's operation. 



TABLE 2-1 CAREER SURVEY PROJECT PARTICIPANTS; NUMBERS SERVED/ 
CHARACTERISTICS 

2-la Participants Characteristics (based upon 1978 project operation): 



AGE 


% OF TOTAL 


RACE 


% OF TOTAL 


1* 


32% . 


Black 


42% 


15 


43% . 


White 


44% 


16 


' 21% 


Asian 


12% 


17 


2% 


Native American 


2% 


18 


A- 1% 


< 




19 


. 1% 


SEX 






> 


Male 


-58% 






* Female ; 


42% 






HANDICAPPED 


13% 



Note: Although the project served youth 14-19 years of age, the typical participant 
was 14-15 years old, economically disadvantaged (70% LLSIL), a freshman or sopho- 
more, and deficient in basic skill areas. 



2-lb Number of CSP Youth Participants 

? ' 1974:- 50 



1975: 108 

1976: 240* , ' 
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1977: 2*0* 

1978: 2*0* . * 
1979: 2*0* •■ * 

* 

f 

TABLE 2-2 CAREER SURVEY PR03ECT COST (BASED UPON 1978 PROJECT 
OPERATION: ONE SITE) ^ 



o Funding Source: Summer Youth Employment Program 

o > * Total Project Cost: $19,726 (Exclusive of participant wages) 

o City of Portland Prime Sponsor Cost: $16,171 (83.3% of Total) 

o Multnomah-Washington CETA Consortium Cost: $2,955 (16.7% of Total) 

o " Cost includes: 10 half-time instructors, 1 full-time coordinator, materials and 

supplies, mileage * 

o • Number of participants: 120 

/ 

o Project Cost per Participant: $164 

o Participant Salaries; $509 per youth (192 hours x $2.65) 

o , Total Youth Participant Wagesi: $61,080 

*The 240 figure reflects project operation at tw,o community colleges: 120 youth per 
site. \ 

No£e:o The City of Portland Prime Sponsor contracts with Portland Community 
College, the project operator.' School District responsibilities, identification/-^ 
referral of youth and academic gcedit provision, are included in a non-financial 
agreement' negotiated .between the City of Portland ^and Portland Public 
Schools. A financial agreement negotiated between the City of Portland and . 
Multnomah-Washington CETA Consortium, the neighboring prime sponsor 
provides*for participation ^f youth frdm the 2 prime sponsorships on a cost- 
.share basis. 
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The Program Development Process ^ ; — ^« 

An analys^of the Career Survey Project's evolution can be pictorially illustrated in a 
general moder(Figure 1-1),. as well as in a more detailed, phase-specific mode (Figure 
1-2). A Model of the Local Knowledge Development Process merely provides a more 
in-depth view of the "Program Development Process." 



The following analysis attempts to describe how we learn about youth programs by 
referencing key stages in the Career Survey Project's development^ and by capturing 

\ 

the systematic, orderly flow of program development proc^sslps to better understand 

t ■ • 

' , Q 

and use the learnings gleaned. 

C^rrei^ Assessment; Policy Analysis and Needs Assessment ' 

The "current youth program ctss^§sment" process emanated from an informal series of 
discussions between prime sponsor anid^^Rooidistrict representatives. The principal 
issues that emerged includedMhe quality of work experience placements made within 
the summer youth employment progtam, and the extremely low enrollment of youtfr in 
vocational/technical courses offered during the academic year within local high 
schools. . * - • A ' >' ' 

' JL 

An analysis of the two issues began. Prime Sponsor and school district representatives 
s 

agreed to approach the examination of each issue by isolating the questions for which^ 
they sought answers: * * 

1. WJiy had previous -efforts to increase the number pf summer work sites 

* > . ' .* " 

providing quality supervision and training been less fruitful than anticipa- 

ted? ' * 
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2. What dynamics contributed to the placement of the more mature 16 and 17 
year-old SYEP population in the most substantdv,e job sites? 

» 

3. ' Conversely, what dynamics contributed .to the recurrent appearance of a Ik 

and 15 year-old "left-over perol", the majority of whom were placed in Jess 
attractive, lackluster work sites, whicK offered little substantive training? 

to* * 

, o How could the f provision of ^ell-structured career experience s~t>e infcreased 

for the l*f-15 yearnold group?, 

© • • 

CO • 

5. Why, after local high schqols had recently increased the quantify arTd 
/ * quality of vocational course, offerings, were enrollment levels in such 

courses frightfully low? „ , 

6. Why wire high school students, already severely deficient in basic* skills, 
opting for college preparation coyY^^ failing, and dropping out? 

a , * • 

" • e < 

\ 7. Why was the enrollment ^community J colleges offering high school 

completion programs with vocational training options increasing so rapidly, 
while°high school, dropout rates were higher- than ^ever? * ^ 

The Prime Sponsor/School, District team began to gather data to assist them in 
concurrently analyzing poficy and assessing needs. Simple, quarftitatively-oriented' 
surveys were conducted to determine the relationshi^l^^ high-srfTobl enrollment 
levels and drop-out rates of local youth for GED completion in technical training 
courses. Interviews with school district career education staff ^_and high school 
students, a qualitative approach, was used to gain a better understanding of why * 
enrollment in high school vocational/technical courses was so low. 
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The data and information gathering process enabled the Prime Sppnsor/School District 
team to conclude that: " ^ 

o ( A substantial number of high school , students who had "dropped out" subse- ^ 

quently enrolled in GED completion/vocational training courses . ; 
o The majority of high school youth had, at most, a limited understanding of the 

career education/vocational training options available to therh , ' ^ 
o Enrollment in school district' sponsored career education courses did not carry 

^sufficient status to entice students^who may have benefited from such training 
o Youth needed to be exposured to the realities of the job markfet ' 
o More effective methods of providing labor market and career information to 

SYEP participants needed to be developed • 
o Well-structured career experiences for 14 and 15 year-old* SYEP participants 

needed to be generated - * * 
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It began to be obvious that SYEP resources should bemused to address these issues* The 
problem identification process enabled the Prime Sponsor/Staff District Team to se1^ 
goals. Then they formulated a statement of purpose: \ • • 



"Tp encourage the student population to make earlier career plans and t^ke 
advantage of ^vocational training opportunities that -may b^commensurate with 
their career interest areas." * 



The statement of purpose* evolved into* a goal statement: "T9 allow Students to 
make better informed career decisions through hands-on exposure to vocation* 
al and technical careers and career information dissemination by attendlhg 
summer exploratory career workshops." ^ 

The knowledge development focal afea had been identified* 
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Selection or Creatiof? of a Model Phase 
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The PS/SEX team conducted a local search for existing modelsu^lthough no such mbdel 
existed within* the Portland Public School system "at that time which could be 
replicated salient features of distributive education and cooperative alternative 
^£$cation approaches were identified. A survey of the local comunity college system 
indicated that career exploration activities of the nature sought Were not in existence, 
nor wqre community colleges serving economicalfy disadvantaged 14 and 15 year-olds. 
A pojl taken of neighboring prime sponsors confirmed the opinion that similarly 
focused projects had not been devised. Although the PS/SD team recognized that 
prime sponsors having similar target populations and geographical characteristics most 
likely had identified similar programmatic needs, the team did not consider tapping 
out-of-state prime sponsors. 

In retrospect, a significant amount of planning/design time could have been saved had 
similar program models been "imported" for replication or adaptation. 

The PS/SD team elected to create a model to m£et the specific needs of their locale. 

Project Goals Analysis 

The PS/SD team initiated the Model Design phase by integrating general systems goals 

identified by the Prime Sponsor for program development with the needs statements 

'* * 

generated by the school district staff. A critical juncture in the progranrTdevelopment 

f m ' 

process occurred when a "goal statement" consensus was reached. Specific objectives 
and pr6ductivity indicator statements were delineated which, the team felt, would be 
indicative of goal attainment. 

Figure 2-3-Career Survey Project Objectives and Productivity Indicators, and Figure 
2-4-Statement of Activities/Timeline for Each Objective, reflect the" outcome of the 
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FIGURE 2-3r_ 

CAREER SURVEY RR03ECT OBJECTIVES AND PRODUCTIVITY INDICATORS 

> 

OBJECTIVES ' . PRODUCTIVITY INDICATORS 

• % s 

1. To increase 120 participants 1 under- 1. - Students knowledge of careers as 
standing of vocational/technicaP , measured by pre-test and post-test, 
careers by providing exploratory ' - Degree of difference between pre- 
education classes in 10 vocational test and post-test, 

areas. - Increase 120 participants' exposure 

^ to job opportunities in ten (10) voca-„ 

tional clusters. 

* * - Increase of 120 participants' know- 

a ledge of- skills, aptitudes, and inter- 
ests required for success in ten (10) 
vocational dusters. 

- Number . of vocational cluster 
areas^attended by each participant. 

- Average number of hours spent per 
participant in each vocational clus- 
ter. 

- Range of hours attended by 120 
participants. 

- Number of youth enrolled at the 
end of the project. 

- Number and types of "hands-on" 
experience provided in each cluster. 
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2. To increase 120 participants 1 utili- 
zation of vocational/technical re- 
sources within the high schools by 
fifteen (15) percent. 

<* * 

3. To increase 120 participants 1 under- 
standing of thefr" obligations as em- 
ployees by requiring attendance at 
sixty (60) percent of the sessions 

1 over the length of the project. 



2. - Number of participants enrolled in 
vocational/technical courses during 
the academic years prior to 1978- 
1979. 

- Number of students enrolling in 
vocational/technical classes after 
participation in the Career Survey 
Project. * 

3. - Attendance records. 

* 4 - Range of hours attended per parti- 
cipant. 



To increase the employment oppor- 
tunities for 120 econpmically dis- 
advantaged youth by employing 
them during the summer in super- 
vised work and instruction activi- 
ties. 

5. To 'maintain effective .leadership 
and administrative management for 
the Career Survey . Project during 
Summer, 1978. 



k. - Total dollars paid to participants. 

- Averge allowance amounts earned 
per participant. 

- Hours of work perf ormed. 

- Total number of youth employed. 



5. - Number and dates of activities 
listed under activities 5.1, 5.2, 5.3, 
etc. 
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FIGURE 2-*: STATEMENT OP ACTIVlTIES/TlMELINE FOR EACH OBJECTIVE 



Objective #1: ^RS-st!ate objective hetfe) 



To increase 120 participants 1 understanding of vocational/tethnologic&l careers by providing explo 
classes in 10 vocational areas. 




cation 



4 



No. Activity 

4 

1.1 Develop curriculum 
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1.2 Hire 10 instructors and 
A secure facilities for 

classes to meet. 

1.3 Develop recruitment cri- 
teria for participants 

I A Recruit participants from 
eligible SPED\ clients 

1.5 Assign instructors 

1.6 Assign participants 



1.7 Recruit team leaders 

i 

1.8 Assign team leaders to 
teams of participants - 

I 

1.9^. Develop orientation f<jr 
team leaders * 



Completion 
Date 

5-31-78 



5-15-78 

5- 01-78 

6- 02-78 
t 

5- 15-78 

c 

6- 15=78 



6-02-78 



6-15-78 
6-15-78 



Measure of completion 
of activity 

Written course outline 
and cdUrse description^ 
of each instructional 
component submitted by 
each of the 10 instructors 
to the City. Courses pro- 
vided as- planned. 

Contract with PCC 



Written criteria 



Participants recruited N 
and enrolled 

Schedule for instructors 
complete 

Participants assigned to 
teams 

Appicants interviewed and 
9 team leaders hired 

Team "leaders assigned to 
teams of participants 

Orientation agenda devel- 
oped ' r 



Other Agencies 
Involved \ 

PCC, HRB 



HRB, PCC 
HRB 

PPS, HRB 
PCC 

HRB, PCC 
HRB, PCC 
PCC, HRB 



PCC^HRBx 
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goal setting effort. The figures provide a sampling of the goal analysis results, which 
were developed for* the 4th cycle of the project Q978). The program operator 
(Portland Community College), as its operational expertise increased, assumed an 
active role in suggesting structural modifications, j * 
* 

The , objectives, productivity indicators, and activity/timeline statements reflect a' 
much more specific, extensive, and quantitative^design than those developed the first 
year* Most of the productivity indicator over-hauling occurred during the second year. 
Formally defined activities and timelines were developed the third year. 

0 

\ 

The refinement of these items was an outgrowth of program experience as the project 
evolved .through the Demonstration, Modification, Process Evaluatiop, In-Program 
Analysis, and Evaluation of Participant Benefits at Program Termination Outcome 
stages. Goals, objectives, and productivity indicators were continually reassessed, re* 
focused, and re-written for the succeeding program cycle. 

< ■ 
Learnings from each modification cycle contributed to an increasingly specific 'and 

more thorough knowledge of which program apjyoaches work best. 

The goal setting and -refinement process was crtitical; it allowed planning, implemen- 
tation, and project assessment to occur, in an orderly, systematic yay. It insured that 
the concept of the project was clarified in atfvancef, and that activities were 
operationally defined* 

Model Demonstration 

The model was implemented via contract with Portland Community College (Summer, 
1974). The majority of administrative energy focused on insuring that the program 
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was initiated and completed (more commonly referred to in yputh programming circles 
as "get, the program up and rolling' 1 ). The assessment was based largely upon a 
qualitative format, but included some quantitative data. The principal instruments 
included instructor questionnaires (narrative responses!, program coordinators 1 project- 
end reports (identification of program strengthsfweaknesses/recommendations for 
change), and team leader project critique sessions. The rationale for the use of such 
qualitatively-oriented instruments is based on the "process evaluation" thinking 
discussed in Chapter, 1 and, in more detail, in Chapter 3. 

In the early stages of program implementation, it is appropriate to concentrate on 
assessing the general workings of the program (the macro-approach) rather than on the 
measurement of outcomes. The focus of the first cycle's assessment was, therefore, 
to develop a 'general understanding of if and how the program achieved its goal. The 
first cycle's assessment was positive. 

In spite" of innumerable "rough edges," it was agreed that the concept of summer 
career awareness/exploratory sessions for fhe 14-15 year-old group was sound. 

Process Evaluation of the Modelt Second Cycle , 

As. the PS/SD team began seriously to consider the long-term utility of the project and 
the transferability of the rotational career awareness concept to other projects, the 
heed to fine-tune the total program process was recognized. 

V 

/ . 

It was agreed that an, adequate lead-time for project planning modification and 
assessment design had to be provided and that suggestions for program modification 
should be solicited from a broader range of education and employment and training 
personnel. 1 ' 



The ranfee of assessment instruments and indicators included: 



Quantitative 

Number of Vocational' Clusters 
attended by each participant 



Average number'of hours 
spent by each participant 
in each vocational cluster 

h 

Range of hours attended 
by the 120 participants 



Qualitative 



o Team Leader Summary Reports 

o Instructor Reports 

o ■ Debriefing Sessions 

o Projecj Coordinator Reports 

o Counselor Recommendations 

o Participant Interviews * 

o Work Experience Coordinator Interviews 



o Number of youth enrolled 
at the end of tfte project 

. o Analysis of participant 
characteristics. 

o Pre/Post Testing x 

o Participant data Cross-tabulation (to check for sex and race bias) 



Outcome Assessment Efforts (Evaluation of Participant Benefits at Program 
Termination/Impact Evaluation of the Effect on the Youth Community) 

To date, outcome evaluations of the project have been limited to very simple, non- 
experimental indications of participant change- at prograni^termination and fpllow-up. 



Participant pre-po$t tests measured career information/awareness gains. Participants 
were polled to determine if project participation encouraged enrollment in high school 
vocational /technical courses. A six-month follow-up of participants was conducted to 
determine how many youth had actually enrolled in vocationally oriented courses upon 
returning to school in the Fall. 9 * 



. 0 
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Although it is normally very difficult to substantiate the impact of a specific project 
on the youth community, institutional change gradually occurred; the community 
college setting is now recognized by the prime sponsor, school district, and community 
college network as a valuable and viable resource to provide services for I/S 
% disadvantaged youth. -Moreover, the rotational" career exploratory concept was 
incorporated if? the design of a year-round YEDPA knowledge development project: 
The University of Oregon Medical Careers Exploration Project. The University 
Wmplex is now engaged in providing 'career^xploratory services to I/S and O/S youth. 



\ 
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CHAPTER III 



EVALUATIVE RESEARCH AS A KNOWLEDGE DEVELOPMENT TOOL 
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CHAPTER 3 
EVALUATIVE RESEARCH AS A 
* . ' KNOWLEDGE DEVELOPMENT TOOL 

* 1 

v 

Program evaluation research is concerned with systematically finding out how well 
Social action programs such as' YEDPA'work. The basic logic of program evaluation 
research evolved in part to address specific questions concerning social program 

development. fc 
1 

It is useful to make a distinction between "evaluation" and "evaluation research." 
"Evaluation" is used in general sense to refer to the social process of making 
judgments of worth. It is a process 'which is basic to most forms of social behavior, 
whether that of a single individual or of a complex organization. Usually an evaluation 
implies a logical or rational basis for making a judgment, but it does not necessarily 
involve the use of a systematic procedure for bringing together and presenting 
^objective evidence to support the judgment^. Thus, the term "evaluation," in its 
common sense, refers simply to the gerieral process of assessment or appraisal. . 

.../'• 

"Evaluation research," on the other hand, refers to the use of social science research 

/ 

methods and techniques for the' purpose of making an evaluation. Thus, "evaluation" 
becomes an adjective specifying a type of research, and the main emphasis- is placed 
on the noun "research." So,. evaluatjpn (or ."evaluative") research refers to procedures 
for collecting, analyzing, and utilizing information which increases the possibility of 
proving rather than asserting the worth of a youth program. A great deal of variation 
in the techniques and subjects of focus is possible within program evaluation research. 
For 1 example, .although much evaluative research fodises -on outcomes^ or results, 
evaluations can also be directed toward program goals and objectives, plan's or designs 



for carrying out the goals, or program activities and processes. As will be shown, thp 
wide range of evaluation methodology allows for adaptability to types and stages of 
youth program development, resources available, and other unique local requirements. 
•Despite this methodological range, a basic framework, or model, of the evaluation 
process can be described.^ In the following two sections, this logical process is 
introduced, then integrated and illustrated in the broader context of the knowledge 
development model introduced in< Chapter 1. 

Program Evaluation Research: A Developmental Sketch x 

The earliest program evaluation approach emerged from the education field. |his 
approach was typically concerned with narrowly determining the degree to which 

program objectives were achi,eve<^. The rationale was thair programs should be 

* . * 

evaluated by comparing the attainment of objectives, usually involving participant 
performance, with clearly specified behavioral outcomes established for the program. 
From such a process, it could be determined if the program was achieving its goals. If 
it was not, prqferam modifications could be made*. This early approach has been 

extremely influential in contemporary social program evaluations. 

t 

T^is approach also has several implications for what program directors and staff must 
know and do. The most important of these is the need for clarity ar^consensus on 
agency and program goals, on methods of assessing participant outcomes, and on what 
constitutes acceptable or desirably participant outcomes. 

Much of current good practice in program evaluation simply expands on this methodol- 
ogy; Perhaps the most important development is the increased concern for process * ^ 

— — — ) 

*This is basically R., Tyler's "Goal Attainment'Model," first described in 1949. 
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assessment in the overall evaluation of ^a particular program. Process evaluative 
research assesses the quality of a topogram during its implementation, and has grown to 
be seen as particularly important during the formative early and intermediate stages 
of a program. One of the principal roles of process evaluative research during this 
formative period is to help discover problems, limitations, and successes in the 
workings of the program and its implementation. 

• A 

Outcome/Impact evaluation is more terminal in nature and is aimed at determining the 
value of the completely developed program - usually in terms of program-related 
changes in program participants. A complete developmental cycle of program' 
evaluation ideally synthesizes both types of evaluation. 

0 . k 

♦ Reflecting the evolution of 'approach touched upon above, many contemporary program 

evaluators now subscribe to a <f-level evaluative process which associates a set of 

evaluation types with a parallel set of needs for planning decisions^ 

& 

Four planning categories may be identified:^ 1 

1. Planning decisions - to set goals and objective^ ^ , 

2. Programming decisions - to" design program operations 

3. Implementing. decisions - to monitor, evaluate, and improve program activities 

4. Recycling decisions -\to judge, learn from, and react to outcomes of the program 
(continue, modify, or terminate) * 



For each class of decisions, a parallel type of evaluative\research activity can be 
identified: * ' , • 



*The following discussion draws froiji Daniel Stuff lebeam (1971). 
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Cohtext Evaluation provides the basis for determining goals and objectives. 

a. Isolate the problems, needs, and unmet opportunities the program can 
* address. ' - 

b. * Search outside the immediate system ,to locate useful values, needs, an^ 

approaches. - ^ ~~ -- 

o Explore research findings ' I 
o Visit other programs 
o Assess community priorities 

c. Examine existing program(s): Wharare the policies, the organizations, - and 
the goals? Are goals being achieved as intended? 

d. Should goals be changed or expanded? 

Input Evaluation provides the basis for selecting program activities to meet goals 

t 

and-objectives. How are the objectives best achieved?- 

a. Identify alternative strategies. 

b. Determine the limitatipns and advantages of each. . 

c. ^ Identify the likely effects and side effects of each approach!, 

d. # Determine what is required to administer and monitor each approach. 
Process Evaluation provides "feedback* about program activities. It. has three 
majpr functions: * y - ^ , 

a. Detect or predict problems. ^ 

b. Provide informational needs for mgnagement/staff decisions. 

c. .Record the procedures. 

/ . . 

Information is obtained and reported as often as the program, staff requires it. 

* ■** ■ 

Product Evaluation provides a measure ancl interpretation of achievemeots^as 

V 

often as necessary. This form ojevaluative research is similar to that identified 
earlier as outcome/impact evaluative research. However,, product, evaluation 
may also assess intermediate attainments within the program. 
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In summary, this approach is designed to assist the. decision-maker at all steps of 
program management. Thus, evaluative research is trfe process of determining the 
kinds of decisions to be made and lessons to be learned,, then selecting^olleoting, andv 
interpreting the needed information in making these decisions and learning these" 
lessons. , . 

Five evaluative research phases, shown in Fig. 3-1, provide a simple framework for the-' 
'decisionmaking, lesson learning process. , > 

V 

Applying An Evaluative Research Framework to the Knowledge Development Process 

The principal reason for introducing a "state of the art" framework of contemporary 

i ' 

•evaluative research is that the logic of this framework can easily be adapted'to fit 
<• . t * ,' * » 

neatly jnto. the original model' of the'lo,cal KD process^introduced in Chapter 1. 

" 5 v ' • ' . ; . ' '\ V / 

■ \ ... a, . • > 

This adaptation is made clWre^qnd .more workable by taking an intermediate- step, 

whereby the evaluative reseaYcn framework (Fig. 3-1) is restated as a flow of decisions 

and activities. ' This restatement is snown as,Figi3-2, which contains all of the original 

framework elements except the "NeeSs Assessment^ block The reason for this 

omission is that, within the KD m<^del (Figure 1-U w^*fiave already concerned 

ourselves with a need/policy assessment earlier in the program development cycle. - 

/ 

A key feature of the evaluative research model (Fig.. 3-2), which we are now inserting 

/ • 

into the overall KD process, is^tftat. we may work our way through the logical flow in 
several different ways. This*f/exibility will meet our/arlier .criterion of allowing the 
suitable adaptation of evaluation approach (methodology>^pe of data, study design, 
data analysis techniques, etc.) for each stage of youth program development. ■ - 



NEEDS 
ASSESSMENT 



PROGRAM PLANNING 

SPECIFYING: 
GOALS-OBJECTIVES- 
ACTIVITIES-OUTCOME 
INDICATORS . 



MANAGEMENT 
INFORMATION 
SYSTEM x 



PROCESS 
EVALUATION 



OUTCOME ic FOLEOW-UP 
EVALUATION 



Needs Assessment ' involves stating the 
objectives. This information is used to 
identify program needs. 



In program' planning, evaluative research can 
provide the project director with tools to*help 
make planning decisions. It can also build 
procedures into the program, that will be 
needed a&sess whether or not the 
•program is operating as planned and how well 
4t is achieving its objectives. 

A Management Information System is essential 
to the monitoring process and to determine 
the extent to which the elements of the 
program, are being implemented as^ planned. 

Process evaluative research provides 
information • about the activities and 
interactions of the program's components in 
meeting program oBjectives. This information 
is used > to make modifications where 
necessary. s 

• ' t: 

Outcome and Impact Evaluative Research 
provide information about the success of the 
entire program.^ This information can support 
a decision to maintain, modify, expand, or 
discontinue the program. 



/ 



FIGURE 3-1: AN EVALUATIVE 




( AMEWORK 



► (Adapted from a model developed by the UCLA Center of Evaluation. 
See Morris and Fitz-Gibbon (1978).) 



40 



t < 



rnnnnAM rt anhino 

GOAl % AHfl 

onjf r tins 




oihc^Hr.jHrAnr 
no am. <^nrtitnt)olonf 



Q»Ann»At>?rl 
oortonr on |M 
inr»Act , |i 



tfAtUAl 
S1WDT 



f commit J 

i 



' "1 




qilAtTt Allft 

nottonr on 

IHrACI 

CWAMJAIIOR 

MOOT 


• 





qit Aft r. 



1 



QtfAIMIIAffnl 

rnoctss . in 

lYAlUAIION J. 
StllOT i 1 

11 




<|UAt. 





qvniu Aitwr 




mqccss 




tfAlOAttoH 




SWOT 






** 




commit 







(MIKohf nn| 
Evninnmr 

lit fOUt. 




(Kit IUTWOWk) 




rtotmr J * ah r»Ai n/uift research Honri 




9 

ERIC 



52 



Let's see how our evaluative research model works when we consider how besl; to 

f * 

assess and learn from a LEA-private sector approach. Assume that, within the broader 
KD process, we have already: 



(a) assessed currently pffered youth programs, < 

(b) ^identified additional unmet youth needs, ^ 

(c) tentatively ident^ie^^D'focus area, and ^ 

(d) determined the proper "learning level," based on how much we know about 
the KD focus area. We have a new LEA-private sector program modej to 
trylDut, and are at learning level 2. 



V 



Nowjjfre are ready to demonstrate the model and, at the same time, analyze what 
happened during our demonstration. At this point ih the overall KD model,- we enter 
the logic, of the evaluative research model (Fig. 3-2). Notice that we could have 

y 

entered the evaluative research model from any of a number of other learning levels; 
but, had we done so, it is most likely that we would work our way through the logical 
flow in a different fashion than exhibited here, where we: 



Needs Assessment involves stating the objectives. This information is used to identify 
program needs. 



1. Plan the madel demonstration, being certain to articulate our . goals and 
objectives i^learly. Clear delineation of program goals is crucial to good 
program evaluation. 



t ■ * 

2. Decide on the most appropriate evaluative research approach suggested by our 

* K . 

goals and objectives. At this rather exploratory level, we might well decide that 



our LEA-private sector link approach .is, best examinedvia a' process evaluation, 
since at this early stage, we are most likely to be interested in capturing a sense 
of the major characteristics of the new linkage's implementation, 

a 

Next we will decide what blend of methods to , use within the process evaluative 

. / 

research approach. At this early stage it is likely that many of the questions 

have about our ney LEA-private sector linkage idea may best be answered by 
using qualitative methods. Thus, techniques such as intensive interviewing may 
be used to explore key questions and then verify their relevancy. For example, 
wh^t are the-^Strious "agendas" of LEA and private sector personnel? *Did 
program participants enjoy the experience? What unanticipated problems or 
benefits were associated with the demonstration? And so on. Of course, at the 
very -least, our monitoring system will also collect certain bottom line quantita- 
tive data on youth participant characteristics arid services delivered within the 
program segment affected by our new model, THis, v and perhaps other quantita- 
tive -information, will allow us to report later on the delivery of the actual 
"service units" that occurred during the demonstration: for example, the number 
of conferences held between LEA and private sector personnel, the number of 

private sector work/exposure situations available to youth participants, as well 

'a 

as participation rates, costs, and so on. 

We would then prepare a final evaluative report (and perhaps a brief inter^ 
report or two) combining quantitative and qualitative findings.- , This report 
serves to document at least the highlights of what we have actually done in 
implementing our new "linkage idea, and what were some, of the effects and 
reactions associated" with our new approach. .At this point, we leave the logic df 
the evaluative research model and ret|rn to that of the broader KD model. 



5. Thevfinal evaluative report Jaecfimes part of the decision and policy making 
process of the local jurisdiction. In addition, the result of our demonstration 
.becomes available to 'others interested in KD. If the results of our evaluation 
-are not utilized , there is little reason to # perform the evaluation in the first 
place. 

°6. Depending on policymakers' decisions, we may next decide to modify the model 
and p&r|orm further, more detailed, and still predominantly process-oriented 
evaluations (learning levels 3 and 4 on the K>p Flowchart). Or, if the initial 
demonstration was judged completely unsatisfactory to meet our needs, we may 
alter our idea or discard it altogether and look for another program approach to 
demonstrate. 

9 e 

As mentioned, different pathways through the evaluative model will be appropriate at 
different learning levels of the KD framework. Generally^ in learning levels 6-8 we 
will tip the methods* balance in favor of quantitatively oriented .outcome evaluations, 
which may involve experimental designs using control or comparison groups. However, 
in such a case, some qualitative material will continue be desirable to' serve as in 
interpretive background for understanding quantitative results. 

It is important to note that as we proceed along the learning levels of the KD Motfel, 
we accumulate information about the program approach we wish to locally implement 
or continue to modify. Indeed, it is the continuous accumulation of information which 
enables us to progress along the spectrum of learning levels. Of course, more readily 
available and generalizable findings from youth programs elsewhere, can greatly assist 
us in this learning journey.. 

The remaining sections of this chapter highlight some 61 the more spjecjf ic principles 



and procedures of the evaluative model as it applies to the youth program knowledge 
development process. * t 

Comments on the Techniques of Evaluative Research 

We have examined the general strategy of knowledge development and inspected the 
role played by the logic of evaluative research in KD.- It now seems appropriate to 
offer further insight into some of the specific techniques of evaluative research. 
Space is limited, and for detailed, procedures the reader is referred to Mangum et al. 
(1976), Sum et al. (1978), and tfbrris and Fitz-Gibbon (1979). The following section 
emphasizes goal analysis, discusses* the balance of qualitative and" quantitative 
methods in youth program evaluation, and offers some useful rules of thumb in 
research design, data collection/analysis, and documentation foi^youth program 
evaluation. 

Goal Analysis 

Clarifying goals enables us to establish a common understanding about what it is we 
want a program to achieve and then to determine how and if the program is achieving 
It. In program planning, operations, and assessment, goal analysis plays an initial role. 

In large part this is because o) the systematic thought processlgoal analysis necessarily 

% • * .... 

subjects us to as individuals and, even more importantly, as a group of individuals with 

different perceptions about the many aspects of youth programming. v 

■■/'-■' • v 

From an assessment standpoint, we can be assured that if we reach no basicconsensus 
about program goals prior to operating a program, no consensus is likely to exist by the* 
end of the program — particularly if the program had proSlems. Thus, from a learning, 
knowledge development standpoint, it is^crucial that operations, planning, and manage- 
ment be involved in ascertaining what it is that we are going to do in the program and 

• / 
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what it is ^that we wish to learn from the program. Although additional program 
actions and learnings may occur, we will have a bottom line agreement against which* 
to judge and orient ourselves, both during and at the completion of the program cycle. 

Techniques of Goa^Analysis * 

Goal analysis is the systematic process of moving from the general to the increasingly 
specific* At each step, there occurs a further honing and refinement of the sub- 
elements of the< original goal. Deal with goal analysis as you would the writing of a 

* 4 

report: first drafts are ' for getting down, not for getting good. Goal analysis 
eventually, lead to polished statements which, in addition to assessment purposes, are , 
also suitable for grant applications and public consumption. 

Five basic steps comprise goal analysis: 

1. Establish the goal . Use whatever words are comfortable, though they may 
be fuzzy and broad. The idea is to "get it down," to get started in the 
overall process. ' Additionally, ~it will be easier to achieve an initial 
consensus on the importance of more vaguely stated intentions. 

Check to make sure the goal describes an outcome rafher than a process , 
so you don't get stuck with the problem of means and ends at the very 
start. For example, *make the goal "understand..." rather than "develop an 
understanding of..." 



*The approach to goal analysis discussed here is that of Robert F. Mager (1972). , 
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2. Jot^down actions that define the goal. Use words and phrases, making no 
attempt to. keep things tidy. Don't try to write objectives - that's for later. 
Two strategics may help you to describe the meaning of the goal: ^ 



a) use the word "by" after the goal. For example: 
GOAL: Increase youth participant services 

BY 

1. Training staff in youth counseling 

2. Contacting schools and juven^ agencies to identify, po- 
tenhal4»rTicipants 

3. Expanding facilities 

4. Planning new intake procedures - - - 



As you try to think of actions to reach the goal, 

^b) Consider whichever of the following seems most helpful: 

1. ' Answer the question, "What would I take as evidence for 
the achievement of the goal?' 1 
" 2. Imagine someone else with the responsibility of* defining 
the goal. How would that person recognize the goal's 
attainment? 

3. Go back over your list of actions and tidy it up. Eliminate duplications and 
items that, upon reconsideration, are not what you want to say. Check for 
abstractions (fuzzies), which can be either discarded or analyzed further. 
You may find making reference to a list of more precise verbs helpful in 
formulating concrete statements. (See Table 3-L). 

•* » 

The following statements illustrate the distinction between "fuzzies" and 
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"concretes". 
FUZZIES 

Be a good counselor 
Defend confidentiality 
Appreciate other people 
Understand participants 



CONCRETES 

1. Name ten counseling competencies 
2% Train staff 

3. Smile when you say hello 

4. Identify fuzzies 



Each of the "concretes" is an action. Naming, reciting, selecting, smiling, 
>and identifying are all actions. The "fuzzies," however, are fuzzy because 
they are abstractions rather than actions. "Being a good counselor 11 does 
not tell what the individual is doing when s/he *is being a good counselor. 
The 'same is true for "defending confidentiality"; how is th6 individual 

defining confidentiality? 

« v » 

m N • 

Write a complete statement for each action. Note the manner anjl/or 
extent of accomplishment that you would insist be achieved before 

agreeing that the goal had been attained. 

- * % 

0 

this complete statement will become an objective . Fully developed 
objectives contain the following features: 

a. " Action - A good objective always states what is expected to be done. 

• • • 

b. Conditions - A good objective always describes the important condi- 
tions (if any) under which the .action is to occur. 
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TABLE 3-i: A categorized "snooping-Hsr' of verbs useful for making objectives more precise 
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c. Criterion - Whenever possible, a good (Objective describes the criter- 
ion necessary for the objective to be considered completed. 

For example; Client services are considered ta be increased when the 
/ program demonstrates it has been able to; 

1. Train two new counselors by June. 

2. Plan hew intake procedures, which will fulfill federal guidelines and 
are considered acceptable to a test group of participants. 

.l : . * - \ 

3. Increase the client caseloacj 50% by 'June. 



4. Expand facilities, provided funding is obtainable, for 25% increased 
* c program usage. 

p < P> 

5. - Identify participa^ populations not previously served, which match 

the results of an independent research survey. 

> 

5» Finally, modify these statements until you can answer yes to the test 
question: '"If something was achieved according to these statements, would 
I be willing to s&y the goal has been accomplished?" When you can answer 
yes, the goal analysis is complete. 



\ 



Goal analysis is an* important part of evaluative research regardless of the KD learning 
level. As is to be expected, the specific learning level will strongly influence the 
nature of the goals we produce. For example, the goal, statements of an initial 
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program demonstration will be quite general, while the refined goal statements of a 
participant outcomes study might actually .t£ke the form of a research hypothesis. 
The main point here is that well-written goals and objectives should point the way'to 
the types v of procedures most appropriate i of evaluating the program. This amounts to 
saying that problem statements and solutipns should be appropriately matched. 

Quantitative yd Qualitative Methodology in Evaluative Research 

In evaluative research, the process of /'problem solution" or "types of procedures" used 
is called the "methodology." Regardless of the specific methodology chosen, several 
basic elements shoukH^e present: « 

•' ^ " \ . . • V 

1. Selecting the basic evaluative approach based on the KD learning level, the 

r ^ 

type of questions asked, the nature of program goals, and who the users of 
the research findings are likely to be. 

2. Developing the evaluative research design. 

3. Collecting materials and data about program events 'and participants as 
called for in the research design. 

* 4. Analyzing and interpreting the findings. 

* » # , • 

5.. Presenting and interp'reting. study results. 

In deciding what sort of methodological approach to use, it is important to realize that 
an almost organic interconnectedness exists between the nature of the knowledge 
development question itself, the method used for obtaining information about the KD 
question, and the data collected according tp procedures outlined within the methodo- 
logy. These three elements interrelate in a systematic way, such that a change in orfe 
of the three necessitates a change in the other two. 

A poor choice of KD methodology can severely limit the kinds of questions we might 
ask and the types of problems studied* Thus, -according to the unique local needs and 
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circumstanaces of youth programs, we should select a research approach which is 
based on the unique characteristics gjid goals of local programs, not an automatic 
evaluation approach using standardized criteria applied across* the board. The task, 
then, is to combine th6 validity of the question with the validity of the answer . 

For example, if a question has to do with determining which one of two youth work 
configurations caused specific changes in the youth participants, a precise answer to 
this outcome evaluation research question will most likely, call for a rather precise 
methodology, possibly involving an experimental research design utilizing a Comparison 
or controfgroup. Data collected within this method would typically be quantitative 
data —that is* numerical results based on test results, questionnaire responses, or some 
• other*sort of quantitatively measurable response. On the other hand,Jf the question 
deals with the nature of participant characteristics and reactions In a demdnstration 
program, oul** choice 6i method might involve intensive interviewing, participant 
observation,, or some other such qualitative methodological approach.* In this case, 
the resulting data from our qualitative metfado logical approach would 4end to be more 
narrative and less numerical in natur^ 

Likewise, the. way in which data is analyzed may*differ considerably from problem to 
problem .and methodology to' methodology. Using quanti^p^e approaches, for 
example, we typically become^ involved in measures of statistical description and 
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statistical inference. However, in the anjaysis t of qualitative data, we seek to 
establish themes and patterns based upon the unfolding results of interviews and other 
material collection procedures.' 

"♦For detailed discussion of qualitative methodology see: "Patton (1980); Bogdan and 
Taylor (1975); Lofland (1971); Cook and Reichardt (1979). 
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It is important to realize that a range of methodological approaches to ; KD exist. 
Currently, the dominant evaluative research approach, accepted almost without 
question by many youth planners and evktastors, involves the use of a natural science 
based, deductive, hypothesis-oriented methodology, which involves quantitative meas- 
urement, experimental design, and presumes statistical analysis to be v the epitome of 
"good" procedure. In v .contrast, the alternative (qualitative) end of the methodological 
scale is derived from the 'tradition of anthropological field study. " • 

The qualitative approach is inductive and holistic; it aims at understanding program 
phenomena 4 through the use of qualitative data, holistic . analysis, and detailed 
description derived from close contact with program participants. Although the 
qualitative approach is sometimes erroneously regarded as being "soft," "fuzzy," and 
generally less desirable than a "harder" quantitative apprpach, a more reasonable 
position is simply that different kinds of problems {i.e., KD learning levels) require 
different types of methodological research approaches. No position on the methodoio- 
gical spectrum is intrinsically better than another. Rather, these are alternatives 
from which the we can choose the solution which best matches our particular researfctr 
problem. * 

It is often said that the distinction between qualitative and quantitative approaches is 
that of "knowing" something, opposed to merely "knowing about" something. This* 
suggests that .qualitative approaches, by which one means those research strategies 
which involve participant observation, .in-depth interviewing, fi€ldwork, and so oh, 
allow the researcher to obtain first hand knowledge about the youth employment and 
training programs in question. Qualitative 3ata collection strategies, then, are aimed 
at describing and understanding^ he-holistic nature of the program component from the 
participants 1 points of view/ 



Using a quantitative approach to youth program learning involves .an attempt to 
establish reliable generalizations about the frequencies of program events and about 
associations among these events. By using a qualitative approach, we can acquire a 
systematic overview of the tissue and fabric .Of program activities, events, or settings. 
It can sometimes be argued that a strongly qjjantitative approach may cause us to 
draw our .attention away from the major features of the ongoing setting itself. We can 
trap ourselves into an exclusive concern with v$ry small problems simply because 
these problems are subject to quantification. On the other hand, a quantitative 
approach is n^s^ssary to provide more conclusive .answers to highly specific questions 
about program operations. Because of »the rigor and control necessary for many 
quantitative evaluative research approaches, the detailed v4ew of the Quantitative' 
researcher may be compared to that of a viewer with a4mall magnifying glass. The 
qualitative researcher, however, has a macroscopic though less detailed, view of the 
total context of the program under consideration. Thus, three important ^are^s- of 
distinction emerge when the tWo techniques are compared: 

1. The scale or scope of program operations which may be identified and 
^, studied; 

2. The perspective .of the analyst (the "insider's" view of the qualitative 
analyst vs. the "outsider's" vieVv of the quantitative analyst); 




3. The methqds and data of the two approaches. 



How do we decide which approach to use? When our research focuses on program or 
participant characteristics, we may well choose a predominatly qualitative approach. 
On the other hand, if our focus is more concerned with the establishment of causes or J 
consequences , we rrfay be predominantly interested in a more quantitative approach. 
For example, in process evaluations of experimental or demonstration youth programs, 



/ 
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a more qualitative approach is often appropriate. Here the a£gumen\ is that not 
enough is yet known about the programmajic approach, selection of target youth 
-communities, and delivery methods to appropriately apply highly specif ic quantitative 
approaches to evaluate the program. The more macroscopic approach of qualitative 
procedures may be fnuch more in order. 



> 



For outcome or impact evaluations of youth programs which may have been estab- 

lished in a prime .sponsorship for a longer period of time and about which a 
f 

considerable amount of information about program delivery methods, problems, etc., is 
already known, quantitative approaches to answering very specific questions may be 
most appropriate. 

* 

Usually, some balance of qualitative and quantitative methods should be incorporated 
throughout the course of an evaluation. Each method can provide a cross check on the 
other. Ideally, evaluative research should tap multiple perspectives through the use of 
multiple research methods in order to capture the most comprehensive view of the 
youth program. Thus, qualitative methods cart provide the context within \yhich the 
quantitative findings can be interpreted and understood. It is q^ite common to obtain 
program evaluation results 'that don't fit with what \^s expected. Wondering- about 
rival explanations of outcomes can Y become very frustrating when there is a lack of 
contextual understanding which surrounds the quantitative assessment. What could 
have precipitated such puzzling quantitative results? Was the measurement instru- 
ment okay? Did the questions asked correctly tap the area being evaluated? Did the 
youth participants percieve these questions in a different manner than was anticipa- 
ted? Such questions can go on and on, and often do; qualitative data that can provide 
the framework for comprehending the quantitative portion of the evaluative findings 
can often provide answers about the larger context. In short, qualitative methods 
often provide the basis for understanding the significance of statistical associations. 
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Notes on Selecting an Evaluative Research Design 

'Most simply stated, a "research design 11 is a plan which states how, by whom, when, v 
and. from whom measurements will be gathered while conducting a knowledge 
development pqtfjett. The initial and most obvious reason for using an evaluative 
research design is to insure a Jwell-drganized study, with all the right people taking 
part in the evaluative research at the right times, A good design, however, provides us 
with something more useful than just keeping the collection of data on schedule. Most * 
basically, it is a way of gathering comparative information about program elements or 
participants or some other set of variables so that results from the program being 
evaluated can be placed in a context that will enable us to judge the program's worth. 

In selecting a specific research evaluation design, the KD analyst should ideally 

consider a number of factors, such as the reliability and validity of outcomes, tfte 

trade-offs in reliability and vailidity which must be m^de due to limited time and 

resources, the specific nature of the youth program or programs being *stu{Jied, etc? T 

*, * • * * « 

Perhaps one of the jjiost important criteria in the selection of a' study design, however, 

*. 

is the "face validity 11 of the research design and of the instruments used within the 
research desigtK Face validity concerns the extent *to whictvan entire study or an 
individual measurement instrument looks as if it measured what it is ^tended to 
measure, A research design and the ins\rumen&^sed within it have face validity if 
youth program decision-makers and information-users can look at the design and the 
items measured by the instruments, and understand whaft is being measured. 

For example, if ke\ youth program decision-makers and informat^n-users have a 
chance to review the items on a questionnaire to be used in our knowlecjge 
development study^as^well listen to an explanation about what the questionnaire items 
were supposed to indicate, they may then be -asked: "If we administer these 
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questionnaires with these- items measuring these, factors, will that tell p^what we 
want to know? Does this approach make sense? Would we believe the results if they 
came oat negative??! It is difficult to get decisjpn-makers and information-users to 
look at designs and instrumentation carefully in advance. But the more this sort of 
review takes> place, the greater the insurance that the study results will be accepted as 
being reasonable by the decision-makers and ihformation-u§ers -and actually be 
utilized. Thus, face validity-is extremely important in research design ahd subsequent 
data analysis, data interpretation, and data utulization. ^ 

As w;ith the face validity of research instruments and the credibility of the research 
design, so, too, cpnsensus on the relevance of research definitions and the units of 
analysiv^re also factors that affect the credibility and usefulness d? Kt). evaluation 
data. Furthermore, activedy*involving information-users in making decisions about 
these issues means more than a one-point-in-tune apSpfcwal of a KD research proposal. 
Ideally, decision-makers and information-users should be involved in all KD research 
decisions as* they occur. Typically, there will be a slippage between the methods as 
originally proposed and the methods as actually implemented in the KD study,. If 4 
decision-makers and information-users , are involved only at the' stage of approving 
research proposals, they are* likely to be surprised when they see a final -report. Thus, 
the making ol decisions about research methods is a continuous process that ideally 
involves checking out changes as they a^e made. - — 

v 1 ° . • ; 

c 

The research design of a KD study will vary considerably depending or/^iether a 
quantitative or qOalitative approach to analyzing, the youth program is used. .Quantita- 
tiye design approaches involve the specific determination of research interests, goafs, 
data collection procedures, and hypotheses well in advance of actually conducting the 

k J 9 

study. Qualitative research studies tend to remain quite flexible from the beginning 

^and well ihto the research. Although qualitative researchers have a methodology to 

follow, the specifics of their approach evolve -as they proceed. \^ 
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KD projects may be quantitatively studied in very simple but effective ways, as well 
as in more complex and powerful wciys: , 

u - 

1. Descriptive evaluation research is concerned mostly with "what is," 
and "how much" and involves the use of descriptive statistics (e.g., 
frequency distributions 0 , measures of central tendency, measures qf 
dispersion) to describe the framework or characteristics of the youth 
program population and environment. It is necessary .to establish a 
conceptual frame of reference, e.g., tjje particular new component 
being studied and its relationship to other youth offerings. Workable 
definitions of measures to be used must also be agreed upon — an 
exercise which should be eased by the presence of well-written goals 
and objectives. v 

At the developmental stage of a KD project, it will probably be 
appropriate % to devote the most fi energy to descriptive strategies 
before moving to the next stages of quantitative study. Thus, the 
descriptive approach wiil be especially appropriate at the earlier 
learning levels of the KD model. 

2. The correlational or comparative levej of evaluation is concerned « 
with relationships among measures developed at the descriptive level. 
For. example, we may determine that work groups comprised of mixed 
income participants do, .indeed, seem -vto compare or correlate with 
higher levels of performance (or attitud&nal change, .etc.) than do 
work groups comprised only of disadvantaged youth. 

tSuch critical examination of relationships anx>ng measures describing 
a youth program may provide valuable insight into what goes on in • ■ 
the program. These insights may then raise meaningful questions, 
which can be^answered by subsequent experimental studies. 

3. The experimental approach to quantitative evaluation research gen- 
erally requires an experimental design in which experimental and 
control groups are compared, and to which program-participants have 
been randomly assigned. The change in some outcome variable of 
interest (such as reading improvement or completion rates) is con- 
trasted between experimental and control groups. 

Experimental designs are often inporrectly assumed to be the best or 
even the only really acceptable type of evaluation methodology. It is 
true, that questions [about program effectiveness (e.g., at learning 
levels 6-8*in the KD model) are often best "answered by experimental- 
designs. Descriptive and correlational approaches may certainly be # 
utilized in such analysis, though less conclusive outcomes will result. 

* * « 

In contrast wj|h quantitative design methodology, qualitative research approaches 

involve a gradual unfolding of the specifics as they procede. Challenging research 

design decisions, which mu§t occur at*the outset of the quantitative approach, occur 
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later in the qualitative approach. Though different in nature from quantitative design, 
the discovery and creation of the design within the qualitative approach can be equally 
challenging. 



A major methodological consequence of the qualitative study of people in a given 

* r 

location is the process of discovery. Of necessity, It is a process of learning what is 
t 

happening. Since a major part of what is happening is provided by people in their own 
terms, we must find out about, those terms rather than impose a preconceived or 
outsider's scheme. It is oqr, the observers', task to find out what is fundamental or 
essential to the people or enviornment in the youth program under observation. 



The same range of design types that exist in quantitative research (descriptive, 
correlational, experimental^can occur in qualitative research approaches. The real 
strength of qualitative methodology lies in its facilitation of discovery. Although 
conjecture regarding causes and consequences oi program events may emerge from 
qualitatiye research, the principal focus of this approach will usually be upon 
de§pFjbing the characteristics of prograr#events and participants. * 

4 

Nonetheless, qualitative approaches may be incorporated into, tor example, an 
experimental study* design. " The forthcoming qualitative results would not be statis- 
tically testable as in the case of quantitative results, but the breadth; comprehen- 
siveness, and sensitivity of the qualitatiye results might well offset this consideration. 
Ideally, an experimental research approach would include both a quantitative and a 
qualitative dimension. The two approaches can complement one another in a very, 
desirable way. 



In qualitative program analysis, we will, of course, have some general questions in 
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mind when we begin our research* These typically fall into, one of two broacT 
categories. 



1. The first category includes questions related to specific substantive issues 
in a specific program setting. For example, in studying a central youth 
employment center, one might examine how youth participants vie\y the 
physical and/or social environment of that center, * # 

2. The second category of interest involves asking questions about topics, 
such as the nature of a certain kind of organization or the dynamics of 
small groups. Thus, for example, a youth participant's experience interact- 
ing with fellow workers in a community home improvement work team 
might be examined," 

It may be helpful to consider different categories which the qualitative researcher can 
examine. Ranging from the small scale to the large scale, one qualitative methodolo-* 
gist has recommended the following spg^trum:* 



1, t ACTS, Action in a situation that is temporary and brief, consuming only a 
few minutes or hours. 



2. ACTIVITIES, Action in a setting of some duration: days, weeks, months, 
constituting significant elements of persons 1 involvements. 



3, MEANINGS, The verbal products of participants that define and^ direct 
action, t 

• / 

4, PARTICIPATION, participant's total involvement with or adaptation to a 
situation or setting under study, # / 

5, RELATIONSHIPS,- Interrelationships among several persons considered 
simultaneously within the program, » ^ 

*Lofland(1971), : ' k 
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6. SETTINGS, The entire setting of a program under study, conceived here as 
the unit of analysis.' 

Further, we may consider within each of these six scal^ units either a static depiction 
of the category under study or phases (or sequences) thrjough which the category being 
studied passes over the course of time. This is sojnewhat like the difference between 
a photograph and a motion picture. 

What distinguishes qualitative methods from other methodologies is that the qualita- 
tive researcher's questions are framed in general terms. Therefore, most users of the 
, methodology try to begin a study without specific hypotheses or preconceived notions. 
To enter a setting with a set of specific hypotheses is to impose preconditions and 
perhaps misconceptions on the setting. 

* s 

During the first days of a qualitative study, we may find that our ideas and areas of 
interest do not fit the program setting. Our questions may nort be relevant to the 
concerns and behavior, of the participants. We will then begin to formulate a new 
research design or new tactics and begin to ask different questions.* 



Notes on Data Collection Procedures 0 

Once the basic approach of a knowledge development project has been determined and 
a research design chosen, data must then he collected. Data collection is sometimes 
considered to be a dirty job in program evaluation (or in any type of analysis for that 



*An excellent example of qualitative research in CETA youth prograr^ is Bonnie 
Snedeker*s Getting There , a monograph published by the U.S. Department of Labor in 
1980. A procursor to this study fas included as part of course training materials in 
the Knowledge Development Workshop series sponsored by the ' Of f ice of Youth 
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matter). For both the qualitative and quantitative approaches, th£ collection of data 
probably consumes the greatest amount of time and effort in the evaluative project. 

It is important that we purposefully decide what to measure. We could, in a youth 
employment and training program, decide to measure an infinite number of things: 
"smiles per second, time scheduled for counseling, types of job placements, number of 
participants' bringing extra materials to classroom activities, self -concept, etc. "To 
measure" is used here in its broadest sense-to record in-order to summarize and report. 
Clearly, we must maintain an intelligent selectivity so that we can concentrate on 
data items of demonstrable value. 

Conducting a Kp evaluation is a matter of gathering evidence to demonstrate the 
attributes and effects of the program or one of its subcomponents. The progranrVs 
learning level position on the KD model, program goals, and the nature of our audience 
will help us ma*ke gross decisions about what program variables to look at. For 
example, we might measure some mix of: 

a. Characteristics of the program context or environment (e.f^ program 
organization, time frame, physical environment, class size, counseling 

style, ^etc). . f 5 

€ ( J / - ~ 

b. Participant characteristics (e.g., age, sex, socio-economic status, language 

ability, skill levels, attitudes, etc.) » 

• * ** ^ 

c. Characteristics of ^program implementation (e.g., activities, interrela- 
tionships, administrative arrangements, principal materials, etc.) 

d. Program outcomes Here, consideration, of program goals and objectives is 
. paramount. Be alert, however, to the results and types of outcomes totally 

unanticipated by planners. - — ~, 
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Beyond such general guidelines, decisions about precisely what information to collect 
will be unique to each^study situation. Every program has unique goals and generates 
unique kinds, of data. Though there is no easy way to decide what data to collect or 
j what variables to examine, here are some rules of thumb which may help:* «^ 

1. Focus data collection where you are most likely to uncover program 
effects, if any occur. 

2. Try to Qollect a variety of information. 

3. Try to think of clever (and credible) ways to detect achievement of 
program objectives (or other participant change). 

U. Focus on the type of information you t'hink members of your audience will 
look for when they receive your report. 

\ 

5. Try to measure things that will enable you to further develop the youth 
program approach, with which you are working, and'thaj will advance local 
understanding of youth programming in general. 

The process of collecting data is considerably different depending on whether we are 

using a qualitative or a quantitative design. For example, although interviews may be 

used in both types of research approaches, we will probably conduct the interview in a 

very different way, depending on the methodology we* are using. As developed in^ 

quantitative research, the activity of interviewing has increasingly been narrowed to 

* • 
force a choice between rigidly formulated alternative answers attached to rigidly 

formulated questions. Here is an example: • 



A Morris and Fitz-Gibbon (1978), p. 20. 
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How interested are you in discovering new vocational opportunities? For 
instance, how often do you use your school's career resource center? 

A. Very often 

B. , fairly bften 

C. not very often 

An interview made up of such questions and utilizing pre-formed categories of answers 
is often called a "structured interview." This more rigidly structured data collection 
activity, so typical in quantitative studies, necessarily assumes knowledge of what the 
important questions are in the KD study and, more importantly, what the main kinds of 
answers to these questions can be. To the extent thai we want to impose our questions 
on others and/or know what is happening with the people we are interviewing, this is a 
legitimate strategy. But to the extent that we do not want to make such an 
imposition, nor assQme in advance that we already know a great deal about our 
respondents' lives, a different strategy of interviewing is. required — a flexible 
strategy of discovery. Qualitative approaches to data collection are typically more 
open and flexible. An example of this is' the "unstructured interview" in which the? 
object is not to elicit choices- between alternative answers to pre-formed questions, 
but instead, to elicit from the interviewee what the researcher considers are 
important questions relative to the KD project. Questions uncovered in this fashion 
may thert ultimately provide the basis for a more structured interview or question- 
naire. - 

Data for evaluative research can come from a great range of sources and be gathered 
by an arsenal of collection techniques. We are limited only by pur ingenuity and 
imagination and, of course, time and money. Some possible data sources are: 

1. # - Questionnaires 

2. Interviews 
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3. * Observation 

4. Psychometric tests of values, attitudes, personality preferences, norms, 
beliefs 

5. Rating by peers, sjaff, or experts 

6. Existing plans, records, contracts, monitoring reports, MIS reports, and 
other statistics within our agency ' a 

% 7. Data collected 'by other agencies (for .example, health records, employment 
security records, etc.) 

8. Tests of information, interpretations, skills,*kpplication of knowledge 

9. Projective tests' 

10. Diary or other personal records 

11. Situational test* presenting the respondent"with simulated life situations 

12. Physical evidence 

13. Documents (such as the minutes of board meetings,- newspaper accounts of 
• program actions, and so forth) 

14. Correspondence 

- * 

A lot of quantitative evaluative research depends on asking people for information. 
Such an approach relies on interviews and questionnaires to gather infprmation about 
youth program participants — who they are, what it is they do in the' program, and 
what their behaviors and^ttitudes are before and after participation in the particular 
program. Frequently the staff can be queried. 

r 

In educationally oriented KD programs, tests are 'commonly used; they can provide 
important data on knowledge and learning. " \ 

, . * Sometimes we can find ways of collecting relevant data by Unobtrusive" methods that 
do not involve asking anybody anything. Thus, for example, the recorded number of. 
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student visits to a career resource center over a period of weeks or months might be 
correlated with actions to encourage such behavior. Sometimes counselor records can 
be used for this, ^n excellent guide, full of imaginative examples and suggestions, is 
Eugene Webb's^ftassic little book, Unobtrusive Measures. Observation can be a very 
important tool for collecting data on both pre-and post program indicators. To 
increase reliability, we should record observations immediately; if observations lend 
themselves to easy classifications, they should be coded on the spot. 

. j- 

Youth program records .and other prime sponsor or subcontractor files are sometimes a 
very good source of evaluative data. Unfortunately, such recordsare rarely as useful 
as they should be. Often a prime sponsor's record keeping and its transfer of intake 
and service information to permanent records tends to be fairly haphazard. Records 
are of t£n inaccurate, out of date, and the definitions and categories used by the prime 
may not be appropriate for youth program evaluation purposes. MIS reports can 
sometimes be very useful, though incompleteness and inaccuracies plague many MIS 
systems. Indeed, record keeping procedures are occasionally changed; and if this 
happens during the period under study, it can torpedo all attempts at before and after 
comparisons*. 

On the other hand, there are compensations for using the prime sponsor's or 
subcontractors records. One is saving the money and time that collecting original 
data requires. Another is the advantage of continuity. Typically, the prime has been 
receiving a continual influx of information on the program. Prime sponsor record 



keeping or MIS systems may not be usable as is - we should check carefully for 

\ 

accuracy and completeness. We may also attempt to change certain standardized data 
collection procedures within our organization, or introduce new items more suited to 
knowledge development requirements. 
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Intensive interviewing and participant observation are the data collection procedures 
which are most common to qualitative program evaluation. Both of .these approaches 
involve u^/ in a more gradual unfolding of an outlook and understanding of the 
participants in a particular youth program.* 

a 

A Comment on Data Analysis and Interpretation 

The nature and organization of the data collected within the research design usually 
dictates the level or complexity of data analysis which is appropriate in determining 
the results of the study. Considerably different analytical tools are used in the 
qualitative approach as opposed to the quantitative approach. Furthermore, within 
each of these approaches a wide range of possible analytical methods is possible. 

The data analysis and interpretation phases of a knowledge development project are 
particularly critical in producing results which are likely to be well-received and 
utilized. It is at this stage that we look at the data and try to make sense outof it. It 
is important to separate analysis and evaluative interpretation. For example, in the 



case of quantitative \analysis, we are involved with organizing the data, constructing 
appropriate statistical tables, and Arranging for the data to be displayed in an orderly, 
usable format. On the other hand, interpretation involves making judgments about 
what these d&ta mean, establishing the implications of the findings, and applying the 
results of our investigation to possible future action. By separating oqt analysis from 

interpretation, it is possible for a variety of users to look at the data analysis alone, 

* 

come to their own conclusions, and make interpretations apart from the judgments of 
the original evaluator. 



^ *Two excellent guides to intensive interviewing, participant observation, and other 
qualitative research techniques are Lofland (197/1), and Bogdan and Taylor (1975). 
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Some, program evaluators strongly advocate the ^valuator's responsibility to draw 
conclusions from data and make judgments about the evaluation results. Others argue 
that the^evaluafor's job is merely to support the data, and that the dedsion-maker 
must make the judgments. (In many local youth staffs, of course, the evaluator may 
also be*a decision-maker). An ideal approach, which is a compromise between these 
positions, involves the early presentation of relatively raw results to decision-makers 
and information users, giving them an opportunity to study and react to the data 
analysis without the intrusion of the evaluator's interpretation. IrT this manner, the 
judgments and findings of key decision-makers and information usefrs may become a 
pant of our own later interpretation of the data. In this way, thp process of Jocal 
utilization of youth program learnings may get under way well before the final report 
is distributed. 

1 

Generally speaking, there are simply no hard and fast rules for interpreting the results 
of data analyses in youth program evaluation nor are there any rules fof drawing 
conclusions from these results. If we have followed a well planned and relatively 



objective evaluation procedure, certain findings may^lerj^ themselves to obvious 
unarguable interpretations. However, other findings may fall into a less clear cut area* 
where alternative and often divergent interpretations are possible. At this point, the 
experience of the analyst becomes very important. In many cases, we must rely on our 
personal knowledge of the specific proprfcim and components which are being evaluated 
and come to some conclusions. Ideally, we will confer with prQgram personnel or 
"significant others" in order to clarify findings. In any case, we should always be able 



to justify and defend interpretations and conclusions on the basis of "our data. In 
addition, it is important to resist having our findings distorted in order to "document" 
the preconceived notions of program or other personnel. 



Three additional rules of thumb for analyzing and interpreting youth program 



evaluation data include: 



Clear, readable, and simplified statistical tables are necessary to enable 
decision-makers and information-users to understand and' interpret the 
data. Sophisticated analyses should be reworded and presented 4n simpli- 
fied and tabular form. 

Data must be "worked with 11 in order to^be really understood. We must 
spend some time getting inside the data. Busy information-users and 
decision-makers will be unwilling or unable to spend days at such a task, 
but a few/fiburs of structur^ time spent by ( an analyst can pay off 
tremendously in a greater understanding of, and commitment, to, the 
evaluation data. 

Evaluators can learn a lot from decision makers' and information users 1 
interpretations of the data by remaining open and listening to what people 
knowledgeable about the program have to say. 

Documenting* Knowledge Development Findings 

A critical missing link in the youth program learning efforts of many local youth staffs 
is simply that of writing down a cogent description of the evaluative study and its 
results. Considerable credit and advantage may be foregone if this last step in 
methodology .is overlooked or discarded. Although we distribu^ "problem" results 
more selectively, there also exists a real need among youth staffs to learrl what 
appears not to work, as well as whai appears to work well. 

A glance at an outline of audiences interested in knowledge development findings 
should provide a sense of. the valdfe of placing study results into a distributable, 
discussable form: * 

* v 1. Prime Sponsor (Staff and Adv^ory Bodies^ 
CETA Prime Sponsor Administrators 
Program PJanners/Evaluators 
Program T$)perators/In-House 
" ' Counseling Staff 

Contractors (LEAs'afcd other) 

Advisory Council , *' 1 

»> • 

' Cit^bouncll , County Commissioners 



.2. 



2. Regional: . 

Regional Youth Coordinator 
Federal Representative 

3. National: 

Office of Youth Programs 

Research/Evaluation Teams 
<f. CETA Prime Sponsors: 

Primes within the State network 

Primes within the National network 
5, Public: - 

Elected Officials 

Community-Based Organizations 

Special Interest Groups 

Press ' 

Private Sector Employers 

Unions 



Here are ways that youth program learnings, if well-documented, can be used by each 
of these groups: 

1. Prime Sponsor (Staff and Advisory Bodies) 

a. To assess the effectiveness of the youth program concept being 
tested in terms, of its benefits to program participants and cost 

4 

effectiveness. 

b. To inform administrators, program planners^program operators, and 
w j counselors of youth programming progress and outcomes. I 



c. To present research project status reports to the-ftdvgory Council. 

d. To present a comprehensive assessment of the learning effort td the 
• Advisory Council. 
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e/ To develop a substantive data base upon which to determine what 

works best for whom under -what circumstances; to re-direct pn-going 

and future youth programming. - ' „ 

• • 

f . To develop a substantive data b^se to use in determining which youth 
^ program pVojecti to delete, modify, expand. t 

g. To help further reiine our knowledge base re; modification of the 
overall delivery system that woufd result in. better provisions of 

♦ services to the client population. 

h. To insure that ICD assessment and evaluation data is utilized. 

i. To provide a data base to analyze the ^delivery capabilities of 
'cdntractors and program operators. 

j. To determine what service mix is most effective in dealing wit(i' 

significant gfoup segments. ' 
Regional Office . * 

a. To meet Regional reporting requirements. 

b. To . inform the Regional Of f lei ofxthe status of youth program 



learning efforts. 1 6 „ 

c. 'To -satisfy the Regional Office Knowledge Development Report 
requirements. , * 4 

d. To insure that written documentation of the knowledge development 
experience is completed to meet Regional As^pssment Reviews. 

National Office & 

a. To jnforrp tfte National Office of successful program models arid* 
innovations. * ^ * 

b. To provide the National Office with a more comprehensive under- 
standing o{ the dynamics of local youth program learning in relation 
ta the overall^ programming eff orts^withjp local j^uns^^tic(hs.%^^ * 

c. To^inform the Nat ional^ Office of -the barriers/obstacles encountered 

in carrying out the challange of youth programming. 4 
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d. To develop a local knowledge base that will assist the National Office, 

* <■ * 

in on-going national policy formulation. s ' 

f t • 

e. Preparation of Annual Prime Sponsor Report on Youth Programs: 

4 

Knowledge Development. 9 e 

CETA'/Prime Sponsor/Program Operators 

a. 'To establish an, informational base upon which to share youth program 

successes and'non-successe*s with other CETA Prime Sponsors* 

* * 

% * 

b. y To provide local* contractors, program operators, and LEAs with 

„ Atfstantive information on local knowledge development efforts. 
Public Consumptioh T 

a. To inform City Council members, Co'unty Commissioners, and other 

elected ,pff icials of youth program evaluation efforts and outcomes. 

* * n • ' " . 

b. To satisfy the increasing demand. for accountability prompted by the 

% - 

T surge of press devbted to CETA. 

c. To use for general public relations (presentations to parent groups, 

*l business associations/communityrbased organizations, news releases, 

* . ° . *' 

' . i ' • ' ' « 

etc.); • 

d. Tp use in soliciting the cooper|tion of local public interest grdups in 

■ ' • ■ 

establishing a solid base of community involvement- and support fpr 

* •* . 9 - - 

- m local youth initiatives. * ,? 
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CHAPTER 4: 

UNDERSTANDING AND USING NATIONAL KNOWLEDGE DEVELOPMENT HNDINGS 
V 

As explained at the outset, this guide is principally intended as a primer on methods of 
knowledge development for local youth staff. Nonetheless, the guide would be 
incomplete without an overview of the national KD effort, which was instigated by the 
passage of the, Youth Employment and Demonstration Projects Aqt of 1977, Many of 
the findings and results of the nation^a&ort are of direct use in planning and 



i^M^or 

operating locat^outh programs, 



A detailed history of the Office of Youth Program's knowledge development scheme 
and corresponding schedule of discretionarily funded projects is beyond our present 
'scope, but we can note the basic rationale and organization of the national KD 
approach. We can also identify the most direct means for local youth staff to explore 
and. gain access to this burgeoning stockpile of useful .youth program findings and* 
results* 

-* " • • ; 

The Learning Focus of YEDf*A , 
Learning about youth programs is the central theme of^YEDPA. The Act was designed 
to "try a variety of solutions to youth employment, problems," as well, as to research^ 
demonstrate, explore, examine, study, and evaluate the solutions tried. 

The Offic£ of Youth Programs (OYP) of the Employment and Training Administration 

was, created, in part/ "to. design and oversee a number of discretionary projects in order 

to answer specific questions generated by the\£as€age of YEDPA, It was OYP which 

coined the term » knowledge development - M a comprehensive^ systematic effort to 

\ * * . 

identify, capture, ctnd replicate the lessons learned about the contributions of different 



services, participation incentives, and program management approaches aimed at 
. increasing the career potentials 'of young Americans." 

• * 

It bears noting that DOL-sponstored efforts to learn more about the employment 
problems of young people are not new. Although attention to youth employment 
problems lessened in the early 70 ! s, such efforts were a major DOL priority in the 
1960's* , . ' 

■ 

. As spelled^out in a series of knowledge development plans, OYP's learning strategy 
comprised a broad array of more than 200 discretionary projects that systematically 

$ addressed critical youth employment issues. For the most part, these projects were 
relatively complex, often involving sophisticated research methodologies, multi-site 
dejigns, and other considerations well out of reach of most local evaluation staffs. * 



Depending on how much was known about a particular youth employment issue, OYP ! s 

discretionary project^ were c one (or a combination of three possible sorts: , , 

** \ 1 

1. Demonstration Projects - an array of activities and program approach configura- 
tions having to* do, for example, with in-school programs, private sector 
initiatives, service mix experiments, educational vouchers, work and education 
councils. ■ ( 

A specific example of a demonstration project ns the Ventures in Community 
Improvement Demonstration.' Among other things, this 2 year project sought ta 
reveal whether it makes .sense to replicate a model community improvement 
project in' widely different areas* This project, successful' in Portland, Oregon* 
was tried in 12 U.S. cities. ■ t 



♦For a synthesis and. review of earlier youth prograxn' efforts, see Mangum and Wali 
(1978), and Walther (1976). „ :* ' . ' 
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2, Special Research Studies ,- designed to provide very basic information and 
analysis .concerning youth employment problems and fo assess the performance 
of programs which seek to better the circumstance's of youth. 



The Youth Labor Market Experience Study* is an example of such a research 
project. This study Analyzes youths 1 labor market behavior 4 to 5 c years after 
graduation from high school. 



3, Program Evaluation - primarily pro^ss and outcome evaluations of programs 
already in place, typically as formula-funded youth programs. ^ 

ft dtk 

V 

The Process Evaluation of Youth Programs, performed by the National Council 

V 

on Employment Policy (NCEP), is an example* of a national KD evaluation 

■\ 

project. It assessed the planning, start-up, and operational experience of YCCl£ 

* - 

and YETP in relation to other employment and training programs for youth. 

♦ • ' \ , ' 

OYP initiated .projects addressing critical youth employment issues in a manner which* 
was thought, to systematically focus new knowledge development'efforts on areas that 

needed attention, building on what was known, and focusing on the next appropriate 

\ ' * 

,» i t . s % - , 

level of inquiry, * . ' f , 

The logical elements of OYP's systematic approach to focys and organize each 
discretibnary project are already familiar to readers of this &uide as the steps 
identified in the lcx;al KD model (Figure 1-2), Of course, OYP applied the logic of the 
KD model on a national, rather- than a local, scale. When OYP accomplished such a 

step in the $D model as "Current Youth Program Assessment" or ^Knowledge Base 

. - * t 

Assessment,^ it was with reference to a generalized, aggregate picture of the youtti 

" • i 9 

employment and training pfqgram needs of the entire nation, * When, the "learning 



levels" assessment within the KD model was applied, often to a multi-site demonstra- 
tion design, the implications of the selection of an appropriate learningjevel held 
firm. For example, although OYP might simultaneously explore a new program rhodel 
at a number oi different sites, the effort * was still typically conceived ks a 
"demonstration 1 ' program, and process-oriented assessment techniques were employed,. 
Correspondingly, outcome/impact research and evaluation techniques were-employed 
in assessing national projects about which more was originally known. ( 

What is significant for local youjh staff is that the admittedly awesome OYP 
discretionary effort can be understood in terms of its many individual projects, each of 
which is examined according to the sarne fundamtrttal KD process described in 
Chapter 1 of this guide. The^bro^d range/of issues examined by OYP virtually assures 
that, the findings of at least certain national projects will be of use to losal youth 
staff. Ipdeed, because of the resources §YP was able to invest in most of its 
discretionary"eff orts. a given set of national findings may prove to Ye among the most 
valuable of "outside resource" considerations made by a locaJ youth staff in conducting 
their^Wn local KD effort/ For example, OYP has typically examined. specific -youth 



program models in a relatively controlled multi-site manner. This approach improves 

\ 

both the likelihood of aapibing* causality to program events, as well as the chances of 

\ 

the model program having b'een demonstrated within a jurisdiction* which more closely 

* \ * ■ ■ * 

approaches that of interested local youth staff. Of course, we can never assume that 

\ ' \ ^ 

"imported" findings will perfectly fit specific local circumstances, but a clo&r lit may 

provide much greater insigh^in developing local programs. 



A Discretionary Study of Vocational Exploration t r 



Examination of the Vocatipnal ^Exploration DemohWat'lpn Project (VEDP) m*y provide 
a sense of the scftpe and rationale of nationaf knowledge development projects. VEDP 

{■ ■ 



is a multi-site test of alternative approaches for conducting a local vocational- 

exploration program. * 

( ' # 

Vocational exploration is an approach to youth programming which is familiar to most 
youth employment and training st^ff. Generally stated, vocational exploration is an 
educational and motivational experience through which youth learn about the world of 
work, look at different occupations,' jobs, and career opportunities, gain experience 
useful in making decisions about future directions, and develop a firm behavioral 
foundation from which to proceed. , 

* 4. A 4 

However 1 , many specific variations and refinements to the basic approach have 
occurred since the earliest vocational exploration efforts in 1971/ Origioally little 

/ " ' ' • • 

more than a work /experience program, various cognitive and affective enrichments 
have come to be seen as essential to the approach. Typically utilized in summ'er 
programming, some .youth staff saw year-round value in the vocational exploration 
approach. OYP's* decision to systematically modify and replicate a number of 
vocational exploration models was, thus, a national outgrowth of the hodgepodge of 
sometimes conflicting information which had accumulated around the basic vocational 
« exploration approach. \ 

Teaming up with the National Alliance of Business, the Human: 

Resource Development Ins titute [(of* the AFL-Clb) , and the St. 

• & . ' ' 

Louis University Center for Urban Programs, OYP . designed. VEDP 

to test alternative vocational Exploration .approaches of varying. 

duration and with different' target groups, . ' . 

' ' * * • •• . 

- * 
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Ttie Vocational Exploration Demonstration Project operates in 16 cities 
throughout the/ country and tests five (5) basic program models which employ 
different strategies for providing a vocational experience for youth. Each 
of the 1(? program operators implement ^a specific program design based on 
ohe' of the following five*models: 

1. Onsite Exposure is occupational exploration throUgh actual 
placement at a private or public sector employer's place of 
business "for worker shadowing, job/task observation and 
rotation, and limited practical ,T hands-on M experience, 

2. Vocational Exposure Laboratory is occupational exploration 
through classroom activities, ''survey style'* vocational 
training 1 , limi|£cl skill instruction and simulated, "labora- 
tory" n*x:k-up work' conducted, at union trade instruction 
institutions, vocational schools, skill training centers, . 

, or community colleges. v * ' 

^ 3. Ecle ctic Exposure is occupational exploration through a multiple 

process approach which combines several basic elements into a 
planned program intervention strategy. .These activities include 
but are not limited to field trips and tours of busin^pses and 
industries, films and seminars given^by .various employers and 1 
K businessmen"* presentation of occupational 'and vocational infor- 
nation, sessions on survival skills, and youth projects such as 
small enterprises. - • ' t • 

4. Multi Modal Exposure provides for the operation of various mixtures 
of the three basic types' described above. v 

5. Extension Model Exposure provides Onsite Exposure ^Mu^ti Modal 
\ * Exposure in a Sequence ^of activities that exten4 over a twelve- 

iftanth "period of time. * > 

• - • ; \ < . f - 



Alljof the local programs have three components- -one serving in-school 

youjh, Another serving out-of-school youtfr, and a third, offered during 

the summer, for both in-school and *out-of-school youth. Within each of 

the| three components, all youth receive the same mix, of^ activities and 

services called for by the specific program model bepjig^bperated. There 

# 

arel four majQr areas that will be investigated through VEDP. These include: 
o examining the Relationships, in a .variety of vocational exploration 
models,* between the people served, the program activities and 
services, the results fcnd the environmental factors. 

comparing summer and r^onsummer vocational exploration efforts. 

v 

comparing the effects of exposure to vocational exploration 
strategies for a 12#month period to similar activities and 
^services offered for shorter periods of time, 
q investigating the effects of vocational 'exploration programs 
upon the attitudes and institutional behavior of business and 
organized Habor. 

This national demonstration will further indicate the^types of youths that 
most benefit from the vocational exploration experience, the specific 
program model that produces the beat results for certain types of yfcuths,-the 
length of the experience that ha^.the most impact on participants, and the 
types of labor market conditions that ar u e best suited to, the five basic 
program models. Local prograrti operators will be able to detefmine the 
amount of program^tic and supportive services that are required to assist 
in the transition to the world of work. The findings will further indicate 



r 



the kinds of implementation strategies that are needed to work with, fhe ~, 

' —t ( * 

private sector in VEDP as it relates to^the size an^types of employers *and 



feie role of the supervisor. Finally, information oji the ease with which 
linkages with LEA' s and' other local actors are obtained in implementing a 
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program, as well as the influence of linkages on the transitional readiness 
of participants will be available. In addition to the evaluations, the 
products will include technical assistance guides, monograph^', and^ 
model management information system. A system is being developed .which 
will provide £or a built-in technical assistance capacity through program 
committees made up of experienced VEDP operators and specialists from 
the National Alliance of Business and the AFL-CIO Human Resources Develop- 
ment Institute* These committees would provide information* on eveiy aspect 
of the program including the design, planning, marketing* implementation, 
administration and research. 

Quite clearly, these results will be important for prime sponsors and 

delivery agents who are considering or have decided on implementing 
/ 

vocational exploration activities. The aggregate findings will suggest 
whether and to what extent vocational exploration can ctfange awarenesses 
and outcomes for youth. The vocational exploration approach will be 
more effective in some settings than others. Prime sponsors and delivery 
agents should pay particular attention to the/relative effect iveness 
demonstration projett sites with similar characteristics to their 
This will suggest whether and what model to implement. Next, the results 
will indicate what groups benefit most from VEPs. The findings can be 
used to establish target group priorities for the projects that are mounted. 
The process studies and how-to-do- it guides, will be useful in the 'imple- 
mentation phase. Prime sponsors or delivery agents might avail , themselves 
of the technical assistance network established under VEDP, tapping the 
expe-tise particularly of nearby sites. ' The evaluations and instruments 
used in the national demonstration might be adopted locally to reference, 
rformance to national norms. In other words, by. careful utilization- of 




the national demonstration findings, local decisionmakers can determine 

o 

whether to implement VEPs, the best, type and target group priorities, as. 
well as getting help in inclement ion and assessment. 
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Finding Out About National KD Results * 

-J 

A range of formal and . informal youth program iriformatibn sources 

> _ * 

have been mentioned 6 in this guide. Short of accidentally happening 

upon someone who has experience with # precisely the type of program 

approach, jurisdiction, and clientele as our own, by far the most 

efficient avenue to youth programming information and assistance in the 

new "information brokerage 11 established under the auspices of OYP's 

Technical Assistance and Training Division. . m 

The function of the youth' information broker is to act as an intermediary in matching 
youth program people, who have questions about a particular youth issue area with 

0 9 #4 * ( 



However, *to rfeduce the number of research variables, a number of program elements 
are maintained in common at all sites; such controls include program philosophy and 
purpose, client eligibility and selection, program reporting formats,' and a number of 
other elements. * <** * 



* 



Many answers about. vocation^ expl&rati<5n"programming are emerging from VEDP; in 
some cases, these questions ire taking the form of more refined questions, that'll 
inquire additional examination. 'While such findings are of value to national youth 
planners in, for example, changing and clarifying youth regulations, they are also df 
enormous potential benefit to j6cal__youJhlstafLjJ}^their efforts to develop and 
implement workable youth programs. The question arises, however, of how local youth 
staff, typically short of time "and rnor^y, can gain access to such findings as ape 
emerging from VEDP. If this question' is not -effectively addressed, there is little 
.chance that local youth staff will directly profit from national knowledge development ' 
findings. 



Finding Out About National KD Result 



A range of formal and informal youft program information sources have ^been 
mentioned in this guide. Short of accidentally' happening upon ,someorfe who 'has 
experience with precisely the type of program approach, jurisdiction, and clientele as 
our dtwn, by far the most efficient avenue to youth programming information and 
assistance is the new "information brokerage" established under the auspices of.OYP's 
Technical- Assistance and Training Division. 

* " * , « , ; [ \ . . 

The f unttiqn of the youth information broken is to act as an intermediary in, matching 

x" . . •'.!*/♦ 

youth program p&ple'who have questions , about a Rarticufcr yOuth issue afea ,wfch 



youth program people who have some answers and experience in that issue aree. For 
example, let's assume that our assessment of ^ocal youth employment and training in 
our jurisdiction suggests that a. vocational exploration program would be 'worthwhile. 
Exciting as the prospect may be, the often conflicting details of how to actually t<} 
begin to implement an approach wiir quickly convince us that some experienced advice 
would be invaluable. We decide to make use of the youth information broker. 

Contact with the OYP youth information broker is' initiated through an authorization 
request to the DOL regional office. After obtaining the necessary authorization, local 
youth staff may telephone the brokerage service directory. Brokerage staff will listen 
to our request, perhaps assisting us in clarifying or refining the typje of question we 
have asked. "In a reasonably short time,we may-expect three types of replies to our 

V 

* \ 

request:^*. ^ v ~~ , * ' * 

a) *the results of a bibliographic search through educational and-employtaent * 
ancf training information banks— including, of course, the latest available! 
findings from OYP's discretionarily funded KD projects! 
% b> a listin^oijjBcoming national, regional, and local training events that are 
ilicable to our Wo^st, * a « 

. . . •* •, ,« 

(^'/^delineation of several negotiable technical assistance options tailored to 



qur^specific/ngeds. • • . 1 , ^ 



1 A t 

ft* 



part'of*tbe broker'^ function is tc^.jguide us to the /nost useful aids among ^wealth 
of (e^sourcfes. that arg^va^ye.ySc? aided, #e ; may quickly^isco.yer s&$e of t^e; 
answers to pur\more , obvious projSfra/n impterrientation questionl*. about vocational' 
exploratipn; such^&s what basis progta^ fnbjiels exist, staffing, budgeting, aW &<*pn- * 
However, we may ^Iso.be alerted Jo a variety of jssues^pgl quesj^ps we may\not yet. 
have asked, about vocational explbratiorv programs^ ^What-abdut the intricacies of 
establishing local', linkages with LEAs arid CBOs 1 ? * .^hat about the -issue of 



remediation among youth participants with widely varied skill levels? bo the needs of 
in-school and oit-of-school youths, differ, subh that different iypeST of vocational 
exploration prograta : approaches are celled for In each .instance? What about 
expectations of private sector employers participating in the, progrann, £nd whit are 

, the best methods for marketing the program concept to these pfeople?^Haw difficult is 

c ' * i 

job site development, and how rnuch-rpjation is it fair to expjgfct,6f certain .types of^ . 

employers as well as participants? What are the key issues and choi£6| in classroom or ^ , 

coping skilft curriculum design? And so on. ^ . • , % < 

Utilizing- Nation^ Findings locally * 

* ' * n ' ' * - * J . ' * 

Our owfi local circumstances, will, of course, be unique. Nonetheless, in implementing 

our vocational exploration program, Sve may. clearly .benefit from the hard* won adv|ce* 

• ' > ? 

of others involved with similar pro j.ec<s. . * 9 v / * ' \\ 



* * 



' In sifting and sorting national findings, ,we are $tone of ttfeMmportant jun^tlpns.thfi\< 

• .« v% ^ ( , > . . ' ^ * * ... W» 

cbnfiects *add Wrtwines local and ' n&tior^rknp^ ledge development eff<yrtV JTfe 

* • * *T • * * ' .* * » ;* * * V » 

, import^ iindlhgs we-<aV- considering &re, themselves, products of the knowleid&e- . 

v*; ? &"hy. ^* - ; .4-; v - 

de^elo^ent ( p^Sces^pSrsUed on learningTtey^ls appropriaTfe toothers' circumstances*. 
We raay 'benefit locajly frpm\these findings, but only to the 'extent that we use themes' 
*-raw= material in. our own version of the same KD process, and at the learnifig level that 
is appropriate nfpr us. "' As'discussed earlier, the particular way in which we respond in** 
our own program development cycle depends on a host of circumstances which defTfie 
"our unique labor market jiovironment, clientele, staff, jurisdic;ion and so on.' We may 
conduct a simple model demonstration-ob a design' Based on an elaborate replication of 
a multi-dimension, model tried successfully elsewhere^ We may design and deliver 
these services with our own staff, or subcontract them to. a CBO or^jother community 
or institutional bqdy\accordipg to our specifications. '.These are Unique actions, but, 

• 85 ' • ' : 




viewed more'broadly, they are actions taken within the overall knowledge development 

process; the fundamental logic of the process holds, regardless of scale (national or 

local), specific learning l?vel, or the sophistication of staff or type of jurisdiction. 

The KD process is a powerful tool; as with any tool, one must learn how to use it with 

» * 

purpose and care,. m * ^ 
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CHAPTER 5 
SELECTED REFERENCES 



The following selection #>f references is not meant to be complete. Insteaj 
represents a thoughtful tulling through the enormous applied social reseapdi litera- 
ture, which may b$ of some help to busy,* often harried, youtfy planne^/evaluators. In 
particular, the literature of social progranvevaluation has blossorp^4 in the past dozen 
years as the field has come o^age. However, much of t^i^litecature^s repetitious, 
and an Attempt has'been made to eliminate redundancy. 



The' references are broken into four categorizes 




ERLC 



L Knowledge D,evelopment/Youth Programs ' * . 

II V Program Evaluation: Employment Related 

III. Program Evaluation: General and Methodological • 

IV. Social Research Methods: General 



I. 



Knowledge 



merct/Yo^th Prograitis 



i; 




e Development. Prepared for U.S. Department of 



and Training Administration-, Office of Youth Pro- 



guide for local prime sponsors; originally prepared for 
> on Knowledge Development sponsorecLby .the Office 



Labor 
grar 

(A lengthly 

. a seri; 

2. Br/ndisf AJniyersity Center for Public Service. Youth Programs: Knowledge 

tHe Youth Initiatives. 



published newsletter.) 

* 88 



"Completing the Youth Agendat^A^lan for Knowledge Development, 
Dissemination, and Application for Fiscal 1980." U.S. Department of Labor, 
Employment * and Training Administration, Office of Youth Programs, 
October, 1979. 

(A compilation of the final national K^D philosophy and discretionary' 
program thrusts of the Office of Youth Programs. Also see earlier plans 
.for FY'78-and FY'79.) ' 

Mangum, Garth, and Walsh. Employment and Training Programs for Youth: 
What Works Best for Whom? 'Washington, D.C.: U.S. Department of Labor, 
Employment and°Training Administration, Office of Youth Programs, May, 
•1978. 

(An extremely useful review Wid synthesis of 17 years of public experimen- 
tation with employment and training programs for youth.) 

Publication List, Office of Youth Programs Library, 601 "D" Street, N.W., 
Room 7108, Washington, D.C 20213. 

(A complete, annotated .listing of all current publications of the USDOL/ 
ETA, Office of Youth Programs. Write for the most current listidjg.) * 

Snedeker, Bonnie. Getting There; A Case Study Report on /the Lives, 
Employment Preparation, and Prospects of YEDPA Participants. Washing- 
ton, D.C, U.S. Department of Labor, Employment and Training Admini- 
stration, Office of Youth Programs, 1980. \ * <\ 

(A clear illustration of the use of intensive interviewing Methods in youth 

program assessment, this excellent work provides eloquent testimony to 

; • \ ... k 

the use and value of qualitative assessment techniques in youth programs. 

/■ • • 

- ~ ' 89 • ; . . 

/ 



7. Vocational Foundation, Inc. You afid Youth. * 

v . — — — . 

(A monthly newsletter devoted yo ^What American business is cjeing to 
employ and train young people.") 

8. Walther, Regis H. Analysis and Synthesis of POL Experience in Youth in 
Transition to Work Programs. Washington, D.C.: U.S. Department of 

' Labor, Employment and Training Administration, 1976. 

, - -r — -* 

^(Good, though now dated, overview of DOL experience" with components of 

employment and training programs for youth. 

# — 

9« "Youth Program Models and Innovations." U.S. Department of Labor, 
Employment and Training Administration, Office of Youth Programs. 
(A monthly series of descriptions of local program innovations distributed 
10/78 - 10/79.) ' ■ . 

Program Evaluation; Employment Related 

1. .Bennett, E.C., and Weisinger, M. Program Evaluation: Af Resource 

v — \ — = ^ — • 

Handbook for Vocational Rehabilitation. New York: ICD Rehabilitation 

and Research Center, 1974. < !• . , 

(A manual intended for vocatioftal rehabilitation program evaluators, which 



is of value to prime sponsor evaluators. ' ■ * 

, » 1 - • .J » IS 

Borus, Michael, E. Measuring the # Impact of Employment-Related Social 
Programs. Kalamazoo, Michigan: W.E. Upjohn Institute v for Employment; 
/- Research, 1979. v ; 1 

(Devoted entirely to a discussion of outcc/me, rather thari process, evaluci- 

♦ • * 

tion*) ' 



90 



3. n CETA: Comprehensive Program Review and Assessment Training." A 
technical training package developed, for the U.S. Department of Labor, 
Employment and Trainirig Administration, 1977. 

(A loose leaf publication which includes many exercises dealing with a full . 

\ ' - .... A / 

'range of prime sponsor assessment "techniques: - orientation, planning 



assessments, monitoring, evaluation, and utilization of assessment informa- 

9 

tion.) ■ ■ k . . • 

♦ ' ' * - 



4. Mangum, Garth, et al. Self Evaluation of CETA Manpower Programs; A 
Guiflg for Prime Sponsors. ^CETA Technical Assistance Document. Wash- 
ington, D.C: U.S. Department oi Labor, Employment and Training Admini- 
stration, 1975. * 

(A practical systematic guide to local m6nitoring and evaluation of all 
CETA titles.) • J S 

5. ^ Sum, And,rew,'et. al. Evaluating the Performance of Employment and 

Training Programs at the Local Level' (Vols. 1-3). Boston: U.S. Dept. of 
Labor,, and Training Administration, Region I, 1978. 

(A very complete guide to evaluation at- the primk' sponsor level with a 
complete discussion and examples of process, outcome, and benefit-cost 
evaluation techniques. An entire volume is dedicated to conducting and 
using the resultsof follow-up studies.) 



III. PrograYn Evaluation; General and Methodological 

1. Cook, Thomas D./ and Reichardt, Charles S., editors. Qualitative and 
Quantitative Methods in Evaluation Research. Beverly Hills: Sage 
. ■ Publications, 1979. 

(A collection of articles which develop the case for using a' blend of both 

• . ■ v ... 

qualitative and quantitative techniques in program evaluation.) 

■ - ' ' ■ ,v' 



* 



2. ■ Epstein, Irwin, and Tripodi, Tony. Research Techniques for Program 
Planning, Monitoring, and Evaluation. New York: Columbia University 
Press, 1977. 1 * ' 

. (A good quick overview of research cpncepts and techniques for adminis- 
trators.) 



Ji „p>ah^:to __JsIew.lYork^- £er ga mmorv Press^- Pub-— 

lished quarterly. f ^ * 

4. "Evaluation Review; A Journal of Applied Social Research. Beverly Hills: 
Sage Publications, published six times yearly. /' 
(One of the principal, multicJisciplinary forum§ of evaluation researchers.) % 

5. Isaac, S. and W.B. Michael. Vlandbook jn Research and Evaluation. San 
- Diego: Edits, 1971. * - ^ 

(A very handy, educationally oriented digest of principles, methods, and 

« * 
strategies useful in all phases of evaluation research. 



t/i* .: 



6. Levitan, Sar A., and Wyrzburg, Gregory. ^Evaluating .Federal Social 
Programs: An Uncertain Art. Kalamazoo Micftfgan: The W. E. Upjohn 

■ ■ V h * ' 

Institute for Employment Research, (1979. _ • " ~ • % . 

(A broad scale assessment of the nature and value of DOL and HEW social 
program evaluation) 

-7. Morris, Lynn Lyons, 'and C.T. Ffez-Gibbon. Program Evaluation Kit. 
Beverly Hills: Sage Publications, 197^. Includes the following books: 
a. Evaluator ! s Handbook 



erJc - • . ; 92 iu % 



b. How to ; ^eal with Goals and Objectives 

c. * How to Design a Program Evaluation 

d. How to Measure Program Implementation 

e. How to Measure Achievement 
i. How to Calculate Statistics 

g. Hpw to Presgr\t an Evaluation Report 



(An excellent set of books rich in procedure and example; examples are 
primarily from the field of education.) r 



8. Patton, Michael Q. Qualitative Evaluation Methods. Beverly Hills: Sage 
Publications, 1980. 

(An dn-depth exploration .of strategies for generating valid, useful, and 

\ 

credibLe qualitative info&mrftion for decision-making.) j 

9. Patton, Michael Quinn. Utilization - Focused Evaluation. Beverly Hills: 
Sage Publications, .1978. • j 
(Combining the theoretical and the practical, Patton examines how and 
why to conduct evaluations. His principal concern is for enhancing the 
likelihood. for actual utilization of evaluation findings.) ' 



10. Rossi, Peter H., ,et. art." Evaluation; A Systemmatic Approach. .Beverly 
' * H'ills: Sage Publications, 1979. ^ * 

\ (A comprehensive text dealing with planning, designing, and implementing 

evaluations of both established and innovative social programs and pro- 
jects.' ' Includes detailed treatment of cost benefit and cost effectiveness 
techniques.) 



© ' • . 93. 7 'V ' ' \ 

ERIC « , ' \ 



/ 



11. Struening,' Elmer Ll and Marcia Guttentag. Handbook of Evaluation 

. • * Research, (Vols. 1 <Sc 2). Beverly Hills: Sage Publications, 1976. , 

« 

(A technical but readable compendium of the methods required at ev^ry 
- step of the evaluation process.* Probably, the 'easiest way to, scan the 
classic articles in the field.) . ^ 



/ 



ERLC 

air I riiTTii m u-*J 



' 12r~ ~Sfuffle5eam, Daniel L. "Evaluation as Enlightenment for Decision Mak-" 
in{*. ,! -An address delivered at the Working Conference on Assessment' 
t Theory," The Association for Supervision and Curriculum Development, * 
Sarasota, Florida, January, 1968. - . 

13. Suchman, E.A. Evaluative .Research. New York: Russell Sage, 1967. 

(Fairly technical and 'academic, this is one of the early classics of the 
I < program evaluation literature.) 

1*. Weiss, Carol H. Evaluation Research. Englewood Cliffs: Prentice-Hall, , 
1972, i * 

(A good basic overview of the purpose and politics of social program 
evaluation.) / 



IV. Social Research Methodology: General 

1. Blalock, H.M. • Social .Statistics (3rd edition). New York: McGraw-rHill, 
1979. ' " * 

(An excellent text, one of the current standards of statistical methods f 
courses in sociology.) s 

2. Bogdan, R., and Taylor, Steven. Introduction to- Qualitative Research 
<y . Methods, New York: 3ohn Wiley* 1975. 

34 \ lUL 



(Good rationale and illustration of t how to conduct and report qualitative 
research.) • ✓ 

Campbell, Donald T : , and\ Stanley, 3X.. Experimental and Quasi-Exp- 
erimental Designs for Research. Chicago: *Rand McNally, 1963: 
(Academic, jthe classic -statement of "quasi-experimental" methods which 
may possibly be used when program conditions will not permit "true" 
exf/erifhental desighs.) h • 

. 1 : ; . — — ; 

Eckhardt^K.W/arld Ermann^M.D. Social Research. Methods: Perspective, 
Theory, and Analysis, New York: Random House, 1977, 
(A basic,- comprehensive research methods text.) 

Gay, L.R. Educational Research: Competencies for Analysis^ and Appl- 
ication. Columbus, Ohio: Charles E. Merrill Publishing Company, 1976. 
(A relatively ^comprehensive introductory text on the methods of educa- 
tional research.) 

Gay, L.R. Educational Research: • Models for Analysis and Application. 
Columbus, Ohio: Charles E. Merrill Publishing Company, 1976. ■ 

» 

(A very basic, self-paced introduction to planning, conducting; and report- 
ing simple educational research. This manual is designed to accompany 
Gay's Educational. Research: Competencies for Analysis and Application .) 

Hays, W. Statistics. New York: Holt, Rhinehart, and .Winston/ 1963. 
(There are many statistics texts, but this one remains an academic classic; 
examples are principally from psychology.) 

Lofland, 3ohn. Analyzing Social Settings: A Guide to Qualitative 

* ' 
. Observation and Analysis.' Belnpont, CArv^adsworth, 1971. 

95 . . ' 







, (A readable gpide to qualitative methodology. Strong on techniques of 






observation and analysis of qualitative materials',.) 
* * * 






Mager, Robert F. Goal Analysis, Belmont, CA: Fearon, 1^972. 

v 






(A short", humorous, and very readable discourse on goal establishment and x * ' 






* ♦ 
clarification.) . - » 

4 . ...... 


•i 


10. 


Nie, N.H. et. ah Statistical Package foe the Social Sciences (SPSS) (2nd 
edition). New York: McGraw-Hill, 1975. 


* 




* 0 

(A tremendous tool to know about SPSS. Is probably tfye l^est of the easily 

, _ ■ \ r 

used "canned" statistical programs.) . " » 


( 






11. 


Stevenson, W. Quantative Analysis in Human Resources Research. Salt 


1 




\ 

Lake City: Human Resource Institute, 1-976. 

0 ' ' « 
(On^ef-t^ie few introductory statistics texts specifically oriented ttpward 

) 

employment and training practitioners.) 

• • ' \ ■ \ • . • 




12. 


V 

Webb, EJ. et. al Unobtrusive "Measures: Non-Reactive Research in the * 
' Social Sciences. Chicago: 'Rand McNally, 1966. • ^ 




\ 


' (A unique explanation of how physical evidence, secondary data, and simple 
observation may be used as alternatives or supplements 'to interviews and 






questionnaires in social science research.) 


* * 


13- 


. " ■ y . . . • ' 

Wright, Sonia R. Quantitative Methods and Statistics. Beverly Hills: Sage 
. Publications, 1979/ 




* 


(A "survival manual" approach intended to serve as a quick and straight- 

* 


"forward overview of the principles and techniques of quantitative social 
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research.) • C 
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